A proximal policy optimization (PPO) agent for optimizing beam bandwidth and power allocation, to improve energy efficiency

Abstract

The disclosure relates to a computer implemented method for executing a proximal policy optimization (PPO) agent for optimizing bandwidth and power allocated to beams established between moving mobile devices (MDs) and a base station (BS), to improve energy efficiency. The disclosure also relates to a network node operative to execute the PPO agent and to a computer readable media storing instructions for executing the PPO agent. The method comprises determining, using the PPO agent, the bandwidth to be allocated to each of the MDs, by predicting data to be transmitted for each MD. The method comprises allocating at least one subcarrier to each MD for a time slot. The method comprises computing the power allocation for each beam, established between the MD and the BS for the time slot, based on the corresponding determined bandwidth and allocated at least one subcarrier.

Description

A PROXIMAL POLICY OPTIMIZATION (PPO) AGENT FOR OPTIMIZING BEAM BANDWIDTH AND POWER ALLOCATION, TO IMPROVE ENERGY EFFICIENCYTECHNICAL FIELD

[0001] The present disclosure relates to beam bandwidth partitioning and data allocation with deep reinforcement learning.BACKGROUND

[0002] Massive multiple-input multiple-output (mMIMO) and millimeter-wave (mmWave) technologies are critical for the future of wireless networks. The wide bandwidth of mmWave, combined with the advanced precoding capabilities of mMIMO, enables significantly higher data rates and allows a single base station to serve multiple users (UEs) simultaneously. However, these technologies face challenges due to the high-frequency bands and narrow directional beams they operate in, especially in mobile networks where channel state information (CSI) changes rapidly. Additionally, conventional implementations of mmWave and mMIMO systems are burdened with high power consumption and substantial hardware costs. At mmWave frequencies, traditional precoding techniques become energy-intensive because they require dedicated radio frequency (RF) chains for each antenna. These RF chains contain power-hungry components like mixers, filters, and analog-to-digital converters, significantly driving up total power consumption.

[0003] Analog beamforming, which uses a single RF chain, offers a promising solution for reducing power consumption, thereby improving energy efficiency, lowering costs, and simplifying implementation. In general, analog beamforming creates frequency-independent beams, though recent advancements have introduced true time delay in each antenna, enabling the generation of frequency-dependent beams. A notable example of this is the “mmflexible” analog beamforming approach of K. Jain, R. Reddy Vennam, R. Subbaraman, and D. Bharadia, “mmFlexible:Flexible directional frequency multiplexing for multiuser mmwave networks,” in Proc. IEEE Conf, on Computer Commun. (INFOCOM), 2023, pp. 1-10, which supports the simultaneous service of multiple UEs by allocating distinct frequency resources to UEs in different directions.SUMMARY

[0004] One challenge is determining, in real time, how much bandwidth and power to allocate to each beam in a mobile environment with frequency-dependent analog beamforming, especially with mmWave channels and mobile UEs. This problem becomes more complex when trying to balance resource allocation with the mobility of UEs, as it involves making decisions based on rapidly changing conditions.

[0005] There is provided a computer implemented method for executing a proximal policy optimization (PPO) agent for optimizing bandwidth and power allocated to beams established between moving mobile devices (MDs) and a base station (BS), to improve energy efficiency. The method comprises determining, using the PPO agent, the bandwidth to be allocated to each of the MDs, by predicting data to be transmitted for each MD. The method comprises allocating at least one subcarrier to each MD for a time slot. The method comprises computing the power allocation for each beam, established between the MD and the BS for the time slot, based on the corresponding determined bandwidth and allocated at least one subcarrier.

[0006] There is provided a network node operative to execute a proximal policy optimization (PPO) agent for optimizing bandwidth and power allocated to beams established between moving mobile devices (MDs) and a base station (BS), to improve energy efficiency. The network node comprises processing circuits and a memory. The memory contains instructions executable by the processing circuits whereby the network node is operative to determine, using the PPO agent, the bandwidth to be allocated to each of the MDs, by predicting data to be transmitted for each MD. The network node is operative to allocate at least one subcarrier to each MD for a time slot. The network node is operative to compute the power allocation for each beam, established between the MD and the BS for the time slot, based on the corresponding determined bandwidth and allocated at least one subcarrier.

[0007] There is provided a non-transitory computer readable media having stored thereon instructions for executing a proximal policy optimization (PPO) agent for optimizing bandwidth and power allocated to beams established between moving mobile devices (MDs) and a base station (BS), to improve energy efficiency. The instructions comprise determining, using the PPO agent, the bandwidth to be allocated to each of the MDs, by predicting data to be transmitted for each MD. The instructions comprise allocating at least one subcarrier to each MD for a time slot. Theinstructions comprise computing the power allocation for each beam, established between the MD and the BS for the time slot, based on the corresponding determined bandwidth and allocated at least one subcarrier.

[0008] The method, network node and computer readable media provided herein present improvements to the way mobile devices bandwidth and power allocation operate.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a schematic illustration of an example system model.

[0010] Figure 2 illustrates a pseudocode of the Newton-Raphson (NR) algorithm.

[0011] Figure 3 is a block diagram illustrating the transformation from power optimization (PO) to data optimization (DO).

[0012] Figure 4 is a schematic illustration of the feasibility of PO vs DO.

[0013] Figure 5 is a block diagram showing an example of how DO explores better the action space.

[0014] Figure 6 is a block diagram illustrating the proximal policy optimization (PPO) for bandwidth and PPO for data (PBPD) procedure.

[0015] Figure 7 is a flowchart of the PBPD procedure.

[0016] Figure 8 illustrates a pseudocode of the PBPD Algorithm.

[0017] Figure 9 illustrates a pseudocode of the greedy bandwidth and channel state information (CSI) for data (GBCD) algorithm.

[0018] Figure 10 illustrates a pseudocode of the greedy bandwidth and optimal power (GBOP) algorithm.

[0019] Figure 11 is a plot illustrating the average energy efficiency (Mb / J) achieved by different resource allocation methods as the total required data increases.

[0020] Figure 12 is a plot illustrating the power consumption (dBW) trends for various resource allocation methods as the total required data increases.

[0021] Figure 13 is a plot illustrating a comparison of slot usage across different algorithms as the total required data increases.

[0022] Figure 14 is a plot illustrating the cumulative distribution function (CDF) of energy efficiency in the deployment scenario compared to benchmark solutions.

[0023] Figure 15 is a plot illustrating the CDF of power consumption in the deployment scenario compared to benchmark solutions.

[0024] Figure 16 is a flowchart of a method for optimizing bandwidth and power allocated to beams established between moving mobile devices (MDs) and a base station (BS).

[0025] Figure 17 is a schematic illustration of a hardware in which steps and / or method described herein can be executed.

[0026] Figure 18 is a schematic illustration of a virtualization environment in which the different steps and hardware components described herein can be deployed.DETAILED DESCRIPTION

[0027] Various features will now be described with reference to the drawings to fully convey the scope of the disclosure to those skilled in the art.

[0028] Sequences of actions or functions may be used within this disclosure. It should be recognized that some functions or actions, in some contexts, could be performed by specialized circuits, by program instructions being executed by one or more processors, or by a combination of both.

[0029] Further, computer readable carrier or carrier wave may contain an appropriate set of computer instructions that would cause a processor to carry out the techniques described herein.

[0030] The functions / actions described herein may occur out of the order noted in the sequence of actions or simultaneously. Furthermore, in some illustrations, some blocks, functions, or actions may be optional and may or may not be executed; these are generally illustrated with dashed lines.

[0031] At least some aspects of the techniques described herein may be implemented using artificial intelligence, which comprises a variety of techniques as would be apparent to a person skilled in the art, including machine learning techniques.Machine learning techniques include deep learning and Neural Network (NN), or Artificial Neural Network (ANN). Both NN and ANN may be used interchangeably herein. In some contexts, an Artificial Neural Network could include biological portions.

[0032] Further, looking forward, in a virtual world (e.g., the metaverse, digital twins, etc.), the techniques described herein could be applied in relevant virtual scenarios.

[0033] Optimizing both bandwidth and power simultaneously in a dynamic mmWave network is complex. This complexity arises from the unpredictable nature of futureinputs and the interdependence between bandwidth and power allocation.

[0034] Using frequency-dependent analog beamforming and given the current and historical CSI of mobile UEs, how can beam bandwidth and power be optimally allocated to each UE to ensure their quality-of-service (QoS) requirements while minimizing overall energy consumption by the end of the optimization period?

[0035] To improve energy efficiency, various research efforts have focused on optimizing traditional beamforming techniques, such as digital beamforming or hybrid beamforming. These approaches typically aim to enhance performance by carefully balancing the trade-off between complexity, cost, and power consumption. However, an alternative line of work focuses on optimizing analog beamforming, which offers the advantage of lower hardware costs and reduced power consumption due to the use of fewer RF chains.

[0036] Most related studies in this area assume codebook-based frequencyindependent beamforming, where pre-defined beamforming vectors are selected from a set (or codebook) and adjusted over time using physical-based techniques like beam tracking. While some recent studies have explored frequency-dependent analog beamforming, these works generally overlook the joint optimization of beam bandwidth partitioning and beam power allocation, which are critical for maximizing the overall system efficiency. Most of these solutions rely on conventional optimization techniques and do not consider the potential of advanced methods such as reinforcement learning to handle the complexity of such a joint problem.

[0037] By leveraging reinforcement learning, the adaptability and data-driven decision-making capabilities of these algorithms can be exploited, potentially achieving more efficient and real-time optimization of both bandwidth and power.

[0038] The present disclosure aims to fill this gap by proposing a novel framework that addresses the joint optimization of beam bandwidth and power allocation in frequency-dependent analog beamforming. The approach presented herein also integrates reinforcement learning techniques to dynamically adjust resource allocation based on the real-time and historical channel state information (CSI) of mobile UEs, thereby ensuring optimal energy efficiency and quality-of-service (QoS) for each user.

[0039] The solution described below aims to maximize the energy efficiency of mmWave mobile networks while ensuring the timely delivery of the required data to all mobile UEs. Since an important goal is to transmit the necessary data to all UEs within a specific period, the problem of maximizing energy efficiency can beequivalently framed as minimizing total energy consumption. In this context, given the input channel state information (CSI) at each time instant, the solution seeks to determine the optimal beam bandwidth partitioning and beam power allocation for each UE, ensuring that all data requirements are met while minimizing overall energy consumption.

[0040] To solve this joint optimization problem in real-time, the Proximal Policy Optimization (PPO), a state-of-the-art reinforcement learning (RL) technique, is leveraged. PPO is classified as a policy gradient method for training an agent's policy network. This network approximates a policy function, used by the agent to make decisions. Essentially, to train the policy function, PPO takes a small policy update step, so the agent can reliably reach the optimal solution. PPO enables continuous adaptation to dynamic environments, making it well-suited for scenarios with rapidly changing CSI and mobile UEs. Instead of directly optimizing the power allocation, an innovative way to optimize the beam-data allocation at each time step is proposed. By simultaneously optimizing both beam bandwidth and data allocation, the joint problem of bandwidth and power allocation is transformed into a simpler and more tractable problem of bandwidth and data allocation.

[0041] This transformation not only reduces the complexity of the problem in terms of action sampling and constraints feasibility but also aligns the optimization process more closely with the actual objectives of the network — delivering the required data to UEs with minimal energy consumption. Through this approach, the solution dynamically adjusts the resources allocated to each beam, ensuring both high energy efficiency and optimal quality-of-service (QoS) for all UEs, even in highly variable network conditions.

[0042] The solution described herein has several advantage and technical benefits, including:The bandwidth and power optimization (BPO) problem can be decomposed into sub-problems and an optimization-based approach can be used to obtain the beam power optimal solution.To enable the proposed BPO network to operate in an online and near-realtime manner with adaptive capabilities, an energy-efficient deep reinforcement learning (DRL) solution is introduced using the proximal policy optimization (PPO) method.The online near-optimal PPO approach ensures compliance with UEs' QoSrequirements and adaptability without further training. It surpasses conventional and hybrid solutions in terms of average energy consumption and efficiency, using at most 6.55 % more power than the optimal solution but at least 38.93 % less power than conventional methods.

[0043] System Model

[0044] A discrete-time model is adopted, in which time is segmented into intervals of length T, where T represents the duration of a 5G subframe, typically T = 1 ms.Optimization decisions are made at each time interval (slot is used instead of time interval in the sequel). T ≥ 1 slots are considered, denoted by T = {1,..., T}. The set of UEs is represented by 풩 = {1,..., N}. A mmWave base station (BS), equipped with a single RF chain and a linear antenna array (while a uniform antenna array could also be used, for simplicity, a linear antenna array is assumed herein) with M antennas, operates at a carrier frequency fcHz. The BS employs a set 풦 = {1,..., K} of K subcarriers to transmit information to UEs whose location is updated at each time instant based on a mobility model.

[0045] Mobility Model

[0046] Referring to Figure 1, the UEs' 10 positions change over time as a function of their speeds and directions. A realistic highway 20 scenario is considered where UEs 10 in different lanes travel at varying speeds, either towards the left or the right. The highway 20 is modeled as a rectangle [xmin,xmax] × [ymin,ymax] with ȳ = (ymin+ ymax) / 2. It consists of six lanes along the x-axis, numbered{L1, L2, ...,UEs in the first three lanes (L1, L2, L3) move from left to right, while UEs in the remaining three lanes (L4, L5, L6) travel in the opposite direction.Mathematically, the direction of UE n is modeled as sign(yn— y), where ynis the y-coordinate of UE n. Let 풮 be the set of possible speeds. For simplicity, it is assumed that all UEs within the same lane Limove at the same speed vi∈ 풮.

[0047] Traffic Model

[0048] C traffic classes are considered, denoted by 풞 = {1,..., C}, each with guaranteed bit rate (GBR) constraints. Similarly to the 5G standard, each traffic class c ∈ 풞 has specific QoS requirements, including: (i) its GBR Qc, (ii) its priority level ec, and (iii) its bit error rate qc. A UE n ∈ 풩 is assigned to exactly one traffic class c, and it is assumed that a many-to-one mapping χ: 풩 → 풞, where χ(n) = c denotes the traffic class of UE n. GBR data for each UE are provided in bits at the beginningof each transmission period that spans T slots. Each UE communicates its requirements to the BS via uplink transmission. The UE n must fulfill its QoS requirements (Qχ(n), qχ(n)) within the T slots.

[0049] In the context of a mmWave mobile network, service delay is defined as the time necessary to fully serve a UE's data request from the moment the request is made until the completion of data transmission. This encompasses all the time spent in the queue, as well as the actual transmission time.

[0050] Signal Model

[0051] The BS employs frequency-dependent analog beamforming to transmit data to UEs. The maximum transmission power of the BS is denoted by Pmaxin each slot. The total bandwidth of the mmWave system is B Hz, divided into K = B / Bssubcarriers, where Bsdivides B and denotes the subcarrier spacing (SCS). The frequency of the subcarrier k G {1,..., K} is determined by fk= fc+ Bs(k — 1 — (K — l) / 2), where fcis the carrier frequency.

[0052] Let sn(t) be the symbol transmitted for UE n at time t, such that |sn(t)|2= 1. The downlink channel matrix between the BS and UE n at time t is denoted by Hn(t) G (CMx K, where hk n(t) G (CMxlrepresents the channel vector between the BS and UE n over subcarrier k at time t. The vector bk n(t)G (CMxldenotes the analog beamforming vector for UE n on subcarrier k at time t. The power allocation matrix for subcarrier k at time t is given by Pk(t) = diag(pk,1(t), ..., pk,N(t)), where pk,n(t) ≥ 0 is the power allocated to UE n over subcarrier k at time t. The precoding matrix for the subcarrier k is defined as Wk(t) =Bk(t)√Pk(t) ∈ ℂM×N, where Bk(t) = [bk,1(t), ..., bk,N(t)] ∈ ℂM×N. Consequently, Wk(t) =[√pk,1(t) bk,1(t), ..., √pk,N(t) bk,N(t)].

[0053] The signal received from UE n at time t over subcarrier k is expressed as—V Pfc,n(0 hk n(t')bk nSn(t') + -y / Pfc.n' (f) ^k,n(J^bk>n'(j^Snr(t') + ^k,n’ (1) where zknis an additive white Gaussian noise with zero mean and power No.

[0054] Problem Formulation and Proposed Bandwidth & Power Optimization (BPO) Design

[0055] Let Kn(t) denote the number of subcarriers assigned to UE n at time t. The beam bandwidth allocated to UE n at slot t, denoted <zn(t), is given byαn(t) = κn(t) / Σ κn'(t) = κn(t) / K (2)

[0056] Given the bandwidth fractions {α1(t),..., αN(t)} and the beam directions, that is, the angle of departure (AoD) {θ1(t),..., θN(t)} at time t, mmFlexible can produce beams at the specified angles while assigning the corresponding subcarriers to each beam. Let 풦n(t) be the set of subcarriers, a subset of contiguous SCs in 풦, that is used to construct the bandwidth part (BWP) for UE n at time slot t, assigned to beam-direction n at time t, given by

[0057] Frequency-Dependent Analog Beamforming Design

[0058] Using the mmFlexible approach described in K. Jain, R. Reddy Vennam, R. Subbaraman, and D. Bharadia, “mmFlexible: Flexible directional frequency multiplexing for multiuser mmwave networks,” in Proc. IEEE Conf, on Computer Commun. (INFOCOM), 2023, pp. 1-10, the BS can generate beams that radiate at specific subcarriers and are oriented to specific angles. This requires as input the number of beams, the bandwidth partitioning among those beams, and their corresponding angles, and outputs the beams radiated at the desired angles and subcarriers. One goal is finding the optimal mmFlexible inputs in terms of beam bandwidth and the optimal power allocation vector so that the total energy consumption of the proposed system is minimized while guaranteeing the minimum requirements of the UEs. An example of the proposed system model 100 is provided in Figure 1.

[0059] The frequency-dependent beams bk n(a(t), 0(t ) is a nonlinear function of the bandwidth partition cr(t) and 0(t), and for simplicity, it is noted as bk n(t). Since UEs’ AoDs 0(t) can be approximated using model-based approaches such as extended Kalman filters, it is assumed that they are known, and the beamforming vector are written as bkn(a(ty). For given beam directions 0(t) and bandwidth fractions oc(t), the designed frequency-dependent beams of L. Lu, G. Y. Li, A. L. Swindlehurst, A. Ashikhmin, and R. Zhang, “An Overview of Massive MIMO:Benefits and Challenges,” IEEE J. Select. Topics Signal Processing., vol. 8, no. 5, pp.742-758, 2014 are used to compute the beamforming vector for subcarrier k G K and UE n ∈ 풩.

[0060] Transmission Protocol

[0061] The transmission power pk,n(t) denotes the power allocated to the UE n using the subcarrier k in the slot t. The beamforming vector is normalized so that ||bk,n(t)||2= 1, ensuring that the power is allocated only to each subcarrier andeach UE.

[0062] At each time t, the BS allocates power pn(t) to each UE n within the interval [0, Pmax]. Each UE n divides its allocated power pn(t) equally among its assigned subcarriers, that ispn(t)∀ k ∈ 풦n(t). (4)

[0063] The achievable data rate (in bps / Hz) between the BS and UE n at time t is given by (hereafter, 1g (•) stands for log2(-)):Rk,n(α(t), p(t)) = lg(1 + pn(t)|hk,nH(t) ⊙ bk,n(α(t))|2 / (N0κn(t))), (5)where ⊙ refers to the dot-product, hk,n(t) ∈ ℂM×1is the channel response at subcarrier k and time t between UE n and the BS, κn(t) = Kαn(t), and N0is the power of the additive white Gaussian noise.

[0064] The downlink achievable data rate (in b / s) of UE n at slot t is:KRn(t) = Σk=1Kζk,n(t)BsRk,n(α(t),p(t)), (6)k=l where ζk,n(t) = 1 if and only if UE n is allocated a subcarrier k at slot t.

[0065] The power consumed by the BS to transmit to UE n at slot t is:Σk=1Kζk,n(t)pk,n(t) = Σk=1Kζk,n(t) / κn(t) pn(t) = pn(t). (7)k=l k=l

[0066] The energy consumed by UE n at time t is:= τ pn(t). (8)

[0067] Problem Formulation

[0068] For benchmark purposes, the design of the proposed intelligent BPO is formulated as an offline mixed integer nonlinear programming (MINLP) problem with the aim of minimizing the energy consumed while respecting the UE's QoS requirements. The MINLP is given by:T N minimize (Pl)subject to fc / n(t) G {0, 1], V k G K, n G N, t G T, (Pla) k.n t) G {0, 1], V k G K,n G N, t G T, (Plb) Kn(t) e K u {0}, v n e N, t e T, (Pic) pn(t) e [0, Pmax], Yn N,t T, (Pld) eKn(t) < pn(t) < Kn(t)Pmax,y n E ^, t E T, (Pie) N^ Pn(t) < PmaxPI t G T, (Plf) n=l NKn(t) = K, V t e T, (Pig) n=l KY ' / k,n(t) = Kn(t),vn e ^r, (Plh) k=l N<k,n(O < l,vfc G %,t G T, (Pli) n=l K< Jfc / n(t) < 1, V n G N, t G T, (Plj) k=l < (h,n(t), V n G N, t G T, (Plk) tk,n(t) ~ (k-l,n(t) < ak,n(t), Y k E K.n E N. t ET, (PH) TT / ?n(t) > Qn(l - qz(n)), V n G >, (Pirn) t=i (An— 6n+ ( 1)T < T, V n G JAT, (Pin) K A t: = 1, Yn G J\T, (Pio) ( k=i K t: ^ (k,n^ = 1 > Vn G N, (Pip)k=i

[0069] The objective function in (Pl) aims to minimize the total energy consumption of the mmWave MIMO system. The constraints (Pla) to (P 1 d) govern the optimization variables, which include beam bandwidth allocation <zn(t) (or equivalently (k,n(t)andKn(l)X power allocation pn(t) and consecutive subcarrierallocation <jk n(t). Constraints (Pie) ensure that in each slot t, UE n is assigned zero power if and only if it is assigned zero beam bandwidth. A small constant e is introduced to enforce that pn(t) > 0 if and only if Kn(t) > 0. Constraints (Pl f) guarantee that the total power allocated to all UEs does not exceed Pmaxat each slot t. The constraints (Pig) ensure that the total bandwidth is properly allocated among all UEs. The constraints (Plh) to (Pli) relate the variables and K, ensuring that the subcarriers allocated to UE n are given by Kn(t) =k=1>k,n( ) and ensuring that the subcarrier k is not allocated to more than one UE. Constraints(Plj) to (Pll) ensure that the allocated subcarriers are contiguous for each UE by relating the variables and a. Constraints (Plm) ensure that the data requirements for all GBR traffic classes are met for all UEs. Finally, constraints (Pin) to (Pip) compute the service delay and ensure that the service delay budget of the traffic class / (n) for UE n is respected.

[0070] The Single Interval Problem

[0071] A challenging aspect of solving the BPO problem is ensuring that the data requirement in constraint (Plm) is met for each UE. This constraint is mathematically represented as:T K& n(t)BsTlg (1 + Pn(t)Akin > Qn,n E N, (9)t=l k=l where- ■(10)and qx(n) = 0 for simplicity.

[0072] Since one aim is to minimize energy consumption, the optimal solution ensures that the constraint (Plm) is satisfied with equality, which means that the required data rate Qnis not exceeded for any UE n. To simplify the problem, consider a single slot t. Constraint (Plm) then becomes equivalent to the following two constraints:K<fc,n(t)SsTlg (1 + pn(t)Akn(a(t))) = yn(t), n e JV, (11) k=lT^yn(t) = l, Vn G JV', (12)t=i where ynt) represents an unknown data volume to be delivered to UE n at time t. Itwill be demonstrated that, with a fixed beam bandwidth partitioning, knowing yn(t) for each UE allows the BPO problem to be solved in polynomial time within each slot. The problem of determining data allocation yn(t) is referred to as the data optimization (DO) problem.

[0073] Assuming a fixed beam bandwidth partitioning, the per-slot (at time t) BPO problem reduces to finding the power allocation for N UEs, denoted by p(t) = [pi (t), pN(t)]T, such that the total power n=i Pn( is minimized andKBsrlg + pn(t)Akn(t)} = Yn (0> Vn e y, (13) k=lwhile also respecting other constraints in (Pl).

[0074] Lemmal. Given the data allocation yn(t), the BPO problem can be optimally solved using the Newton-Raphson (NR) method, producing a unique power value for each UE n.

[0075] Proof of Lemma 1. Given the data allocation yn(t), the BPO problem can be optimally solved using the Newton-Raphson (NR) method, producing a unique power value for each UE n.

[0076] Let / n(Pn( ) be defined as follows for pn(t) G [0, Pmax]K fn(PnCt)) = BsTlg (1 + pn(t)Akn(t)) - yn(t). (14)k=l

[0077] Taking the derivative of / n(Pn(0) with respect to pn(t) yields= k=l(1S)indicating that / n(Pn( ) isanincreasing function of pn(t). Given that fn(0) = — yn(t) < 0, the power optimization problem is feasible if and only if fn(Pmax) 0. Assuming fn(Pmax) > 0, thus / n(Pn( ) has a unique zero for each UE n. Applying the NR method allows to efficiently determine the power allocation for each UE n. In the BPO problem, a sum-power constraint should be ensured in each slot:Sn=i Pn(0Pmax- To achieve optimal power allocation using the NR method, a straightforward approach is to apply the NR method N\ times for all possible UE permutations. Since / n(Pn( ) has a unique zero, the order of UEs does not affect the solution as long as n=i Pn(PmaX- If the solution results in n=i Pn( > PmaX, the power optimization problem becomes infeasible, necessitating either: (i) reducingyn(t) to satisfy all UEs, or (ii) retaining yn(t) and maximizing the number of UEs that meet their GBR requirements through an admission and power control mechanism as shown in Figure 2.

[0078] The idea behind the solution presented herein for the power allocation is to transform the power allocation problem to a data allocation problem based on Lemma 1. A flow chart of the proposed slot-based approach is shown in Figure 3 and for clarity, the NR method is shown in Figure 2. Following Lemma 1, the bandwidth and power optimization (BPO) problem is transformed to bandwidth & data optimization (BDO) problem.

[0079] The bandwidth & data optimization (BDO) Solution: A Deep Reinforcement Learning Approach

[0080] The transformation to solve the BPO presented previously can solve the problem knowing first the bandwidth partitioning and then the data allocation at each slot. At each slot, the DRL agent, makes a joint decision of bandwidth and data and then receives a feedback signal as a reward. By continuously interacting with its environment, the DRL agent can effectively learn an efficient trade-off between data delivery and energy to respect the UEs requirements. To do so, first a Markov decision process (MDP) formulation is provided. The MDP is defined by the tuple (. S', c / Z, 51, 53), where denotes the state space and c / Z represents the action space. The function l:< S' X cZZ x < S' i-> IR provides the expected immediate reward that results from the transition from state s G to state s' G after choosing the action a G c / Z Lastly, the function 53:< S' X cZZ x < S' i-> IR is a probability transition function that, given the triple (s, a, s'), returns the probability of transitioning from state s G to state s' G. S' after choosing action a G Jl. Each element of the MDP tuple is defined in detail in the following sections.

[0081] State Space

[0082] Definition 1 (State Space): The state space S fi) includes all relevant information that can be collected by the BS at slot t. It is represented by a collection B(fi) = {Bftf...,£?5(t)} of five matrices Bft) G [0, l]Si, where = H X N is the dimension of the matrix Bffi) for i G {2, 3, 4}. The first matrix has dimension S = H X 2N and the fifth matrix has dimension S5= H X 1. Here H > 1 represents a history window during which the previous information is collected by the BS.

[0083] In the slot t, the state space has dimension S = (61V + 1)H. Before describing the content of each matrix Bj(t), their general structure is presented. Every matrix Bt(t) of the state space tracks the previous H recent pieces of normalized information stored in a moving window-like structure. Specifically, it always keeps the most recent information at the last row of every matrix and shift all H — 1 previous values up by one row. Mathematically, at time t > 1, the hth row (for h G {1, 2,, HJ) of Bt(t) is given by the corresponding pieces of information of the UEs at time max(t — H + 1 + h, 0), where by convention the information at time t < 0 is given by the zero vector 0 G [0,1]X'xl, where Xt= N for i G {2, 3, 4], X4= 2N and X5= 1.

[0084] £?i(t) = [0, l]Hx2Nrepresents at time t > 1 the normalized (x, y) positions of the N UEs during the previous H slots.

[0085] =[0 l]HxNrepresents at time t > 1 the normalized average CSI of N UEs during the previous H slots (the average CSI is performed over subcarriers and antennas).

[0086] B3(t) = [0, l]HxNrepresents at time t > 1 the normalized data requirements of the N UEs during the previous H slots.

[0087] B4(t) = [0, l]HxNrepresents at time t > 1 the normalized speeds of the N UEs during the previous H slots.

[0088] B5(t) = [0, l]Hxlrepresents at time t > 1 the normalized time remaining until the end of the optimization period T.

[0089] Note that normalization is performed using a Min-Max scaler. A state in a slot t can be written as s(t) = [Bfitf B2t), B3t), B4t), Bst)fi.

[0090] Definition 2 (Satisfied UE): A UE is satisfied if its QoS requirements are met according to its packet error rate and packet delay.

[0091] Definition 3 (Terminal State): A terminal state occurs when all UEs meet their QoS requirements.

[0092] Definition 4 (Truncated State): A truncated state occurs when the current time reaches the end of the time horizon T while the UEs are not satisfied.

[0093] Definition 5 (Episode End): An episode ends if and only if the current state is either terminal or truncated. That is, an episode lasts at most T intervals, i.e., it can last fewer than T intervals if the DRL agent chooses actions that can satisfy the UEs earlier.

[0094] Action Space

[0095] Definition 6 (Action Space): The action space Jl(t) in the slot t is given by the set [0,1]2N, where the first N elements represent the beam bandwidth allocation of the N UEs and the last N elements represent the beam power allocation of the N UEs.

[0096] In principle, at slot t, an action of the BPO problem should be given by a(t) = [a(t),p(t)] e [0,l]wx [0, Pmax]w. (16)

[0097] From this action, one can obtain a feasible solution (the feasibility of the solution will be discussed in Remark 1) to the BPO problem where the variables (t) and K(t) and the variables corresponding to the contiguous subcarriers cr(t) can be obtained from cr(t) and equations (2) and (3). Nonetheless, the proposed method is to not optimize power directly. According to Lemma 1, the DRL agent looks for the data allocation Qn(f) for each UE n in each slot t instead of looking for pn(t). Given Qn(t), the DRL agent then applies the NR method to obtain pn(t). However, for better training (as will be seen in the experimental results in Table III), it is a good strategy to normalize the action chosen by the agent. Therefore, the action space is normalized to [0,1]2Nas given in Definition 6, and thus a vector a(t) G cZZ(t) is normalized and is divided into two parts:The first N elements a1: / v(t) correspond to the beam bandwidth partitioning [<z1(t),...,a / v(t)]AT such that an(t) = crn(t) for 1 < n < N,The last N elements O. N+I-.2N.^ correspond to the GBR data allocation [Qi( < ■■■< QN( ]AT such that an(t) = Qn(f) / Qnfor N + 1 < n < 2N.

[0098] The DRL agent might choose an action a(t) G cZZ(t) that is not feasible according to (Pl)'s constraints. The main two feasibility issues are the bandwidth partitioning should sum to 1 and a zero bandwidth should imply a zero data allocation. That is, if the agent chooses a(t) such that an(f) A 0 for all n G N*, then Snejv*an(=1 and Qn(0=0 for all n g N*. Thus, the agent needs to first clip the chosen action to lie in cZZ(t) (if it happens to be not in cZZ(t) due to some random exploration noise) and then apply a softmax function to the first N elements to make sure that the bandwidth partitioning sums to 1. Finally, it must check whether ®n(0 = Qn(0 whenever an(t) = 0 or Qn(t) = 0.

[0099] Remark 1 (Relaxed Feasibility with DO): Ensuring the feasibility of actions becomes more complex when the agent directly performs PO. When Bandwidth Optimization (BO) and PO are performed, the agent must ensure the feasibility of both parameters during action sampling. For PO, the agent must respect theconstraints (Plf). For BO, it must also respect the constraints (Pig). In contrast, when DO is performed instead of PO, the agent does not need to enforce any constraints related to data allocation during action sampling.

[0100] An illustration of the feasibility issue is also shown in Figure 4.

[0101] Remark 2 (Simpler Exploration with DO): Consider a scenario with a single UE requiring Q bits of data across two timeslots, each with the same CSI. Assume that 0.5 watts can transmit 2Q / 3 bits and the maximum power available, Pmax, / s0-5 watts. For fairness between PO and DO, let the power and data allocation in each slot be chosen from the set {0, 0.5}. In PO, to meet the UE's data requirement, p1= 0.5 watts is allocated in the first slot and p2= 0.5 watts is allocated in the second slot. This results in a total power consumption of Ptotai= 1 watts. In DO, by allocating = 0.5 for the first slot and y2= 0.5 for the second slot, the UE's data requirements are met more efficiently. Since, by assumption, 0.5 watts deliver 2Q / 3 bits, delivering Q / 2 bits requires less than 0.5 watts in each slot. Therefore, the total power consumption for DO is Ptotai< 1 watts. This example demonstrates that PO requires a more complex exploration of the action space, such as considering additional power levels.

[0102] An illustration of Remark 2 is shown in Figure 5 with Q = 150 bits.

[0103] At each step of the PBPD procedure illustrated in Figure 6, the agent 600 observes the environment and updates its state (Step 1), which is processed by a deep actor-critic neural network 610 (Step 2). The actor 620 samples actions (Step 3) based on a squashed Gaussian distribution. The critic 630 evaluates the current state. After collecting sufficient transitions (Step 4), policy gradient descent updates the policy (Step 5). Upon reaching a terminal or truncated state, the environment resets and training continues.

[0104] Given a state s(t), once a feasible action a(t) is chosen by the agent, the environment transitions to a new state s'(t) and a reward signal is fed back to the agent.

[0105] Reward Function

[0106] The reward function 3l(s(t), a(t),s'(t)) is designed to reflect the objective of minimizing total energy consumption as defined in the optimization problem (Pl). Since the goal is to minimize energy consumption when all UEs have met their requirements, the reward function is structured to provide feedback onlywhen a terminal state is reached. Specifically, at each slot t, the agent receives a reward of 0. However, upon reaching a terminal state (as defined in Definition 3), where all UEs satisfy their QoS constraints, the agent receives the following real- valued reward:r 1 vt — VN - if is aterminal state p(t) ±, (17)— C, if s(t) is a truncated statevQ, otherwise

[0107] Here, Sn=i Pn( denotes the total power consumed by all selected (a UE is considered selected by the agent if its action is non-zero.) UEs at time t.

[0108] Remark 3 (Reward rationale): If all UEs have their data requirements fulfdled at any time slot, the reward is inversely proportional to the total power consumed so far by all UEs. If the agent reaches the last time slot (t = T) and some UEs still have unmet data requirements, the reward is —C, where C > 0. This penalty reflects the failure to meet all data requirements by the end of the episode. If at any time t < T not all UEs have their data requirements met, the reward is 0, as the goal state is not reached yet. This formulation balances the efficiency of power usage with penalties for unmet data requirements. If the agent avoids the —C penalty, it indicates that there is usually enough time to adjust allocations and meet all requirements, even if some UEs are not satisfied in earlier slots due to variable CSI conditions.

[0109] In a terminal state, where all UEs have met their QoS traffic requirements (as per Definition 3), the number of satisfied UEs is guaranteed to be N. If the agent reaches the end of the episode without satisfying all UEs, it transitions to a truncated state (as in Definition 4), receiving a reward of (t) = — C and the episode restarts. The parameter C is a hyperparameter used to penalize the agent for not satisfying the UEs after the T slots. This prevents the agent from learning that a reward of 0, which occurs in nonterminal states, is preferable, thus ensuring that all UEs are eventually satisfied.

[0110] The agent aims to maximize the reward function (t) as the training timesteps progress. Ideally, the agent reaches a policy where it can satisfy all UEs1QoS requirements while consuming low energy.

[0111] Transition Function

[0112] The transition probability function ^3(s(t), a(t), s' (if) is a complexstochastic function that characterizes how the system evolves from one state to another based on the current state s(t), action a(t), and resulting state s'(t). It is influenced by factors such as the underlying mobility model, changes in the CSI over time, and the chosen action. For example, if the action at a given timestep partitions the bandwidth to satisfy all UEs, the system will transition to a terminal state immediately.

[0113] Remark 4 (General Transition Function Expression): The general form of the transition function can be expressed as:4J(s(t + 1) = s' I s(f) = s, a(t) = a)= qB(B1(t + l), B2(t + l), B3(t + l), B4(t + l), B5(t + l) | B1(t), B2(t), B3(t), B4(t),a(t) ). (18)

[0114] Given the complexity of the proposed wireless communication system, the analytical form of the transition function is highly complex and depends on various stochastic subprocesses, whose general structure for the transition function in the MDP is^3(s(t + 1) = s' I s(t) = s, a(t) = a). (19)

[0115] Specifically, for the BPO problem, the state s(t) is composed of several components including UE positions, CSI, remaining data requirements, etc. The action a(t) includes beam bandwidth allocation and power allocation (obtained through the data allocation using the NR method). Thus, the transition probability includes: (i) the transition probability for UE positionsgl(B1(t + l) | B1(t),v(t), S(t) ), (ii) the transition probability for the average CSI at time t + 1, which can be modeled as a function of the current average CSI and UE positions:gl(B2(t + l) | B2(t), B1(t + l) ) = gl(B2(t + l) | B2(t), B1(t),v(t),<5(t) ), (20) (in) the transition probability ofUE speeds ^3(B4(t + 1) | B4(t), B1(t) ), (iv) the transition of the remaining time which is deterministic and decreases as time progresses befit + 1) = b5(t) — where b5is the most recent value in Br and theremaining values are obtained by rolling them up by one row as discussed previously.

[0116] The closed-form expression of the transition function is not available; thus, the model-free reinforcement learning approach can be used to address the Markov decision process (MDP). In particular, the agent can learn the structure and dynamics of the transition function through experience, allowing it to perform effectively under varying network conditions.

[0117] The PBPD Algorithm (PPO for Bandwidth and PPO for Data)

[0118] The main operations of the PBPD algorithm are shown in Figures 6 and 7. In Figure 6 the PBPD algorithm is shown as an MDP where the agent (the BS) that implements PBPD is trained through interacting with its mmWave mobile environment. In Figure 7, the basic flowchart of the PBPD algorithm is presented.

[0119] Figure 7 illustrates a method for training the PPO agent. The method starts, 701, and enters a training loop, 702, at timestep 0. If the max training timesteps is reached, 703, the PPO training ends, 704. After entering the training loop, the environment is reset, 705, and the episode loop, 706, is entered. A first state s is observed, 707 and an action is selected, 708. Bandwidth partition and data guesses are produced at this stage, step 709. The Newton-Raphson method is applied, 710. If the total allocated power is greater than the available power at the base station, decision 711, the admission and power control method 712 is applied to reduce power allocated to selected UEs and then the method moves to step 713. If the total allocated power is smaller or equal to the available power at the base station, then the method moves to step 713, where the next state is obtained, and the reward is computed. The trajectory (state, action, next state, reward) is then stored, 714. If it is time for a PPO policy update, 715, the PPO policy is updated, 716, otherwise the method continues at 720. If it is time for a standard deviation action decay, 717, the decay action is taken, 718, otherwise the method continues at 720. If the episode is done, the method continues at 705, otherwise the episode and timestep are both increased by 1 and the method continues at 703.

[0120] The core idea of proximal policy optimisation (PPO) is to alternate efficiently between data sampling for environmental exploration and applying stochastic gradient ascent to optimize a carefully designed surrogate objective function. Unlike classical policy gradient approaches that perform a single gradient update per data sample, PPO allows for multiple epochs of minibatch updates by introducing a novel objective function. Although PPO inherits some benefits from its predecessor, trust region policy optimization (TRPO), it is simpler to implement. The PPO-based deep reinforcement learning approach is illustrated in Figure 6. This method is referred to as PPO for beam bandwidth optimization & PPO for beam-data optimization (PBPD) to solve the BPO problem. PBPD aims to learn a policy function 7T: cZZ(t) X S(t) -> [0,1] that generates a probability distribution over the optimal actions [cr(t), y(t)]T(bandwidth and data allocation) given the current state s(t) andthe policy parameters λ. PPO iterates over E episodes, where an episode ends either when the time horizon T is reached or when all UEs are served, whichever occurs first. The current timestep in PPO is denoted by t, and it is presented as a pseudo-code in Figure 8.

[0121] Referring to figure 8, the PBPD algorithm initiates by collecting a set of trajectories, each consisting of sequences of (state, action, next state, reward) generated through the policy π(a|s(ι); λ). This policy is implemented as a Deep Neural Network (DNN) 610 parameterized by λ. Thus, the PPO policy can be represented as π(a|s(ι); λ), functioning as a non-linear function approximator modeled as a DNN within the actor-critic framework. For a given state s(t) at timestep t, and its corresponding weights λ, the DNN outputs the action a(t). The PPO policy before and after the updating process is differentiated. The update process occurs every futimesteps, i.e., whenever ι mod fu= 0, where ι denotes the current timestep. The policy before the update is referred to as πold(·) and the policy after the update is referred to as π(·).

[0122] The actor network (ActorNet) 620 and the critic network (CriticNet) 630 are parameterized, respectively, by λactorand λcritic. Both ActorNet and CriticNet can be expressed as compositions of nested functions. Let fℓactor(s; λℓactor) and fℓcritic(s; λℓcritic) denote the functions of ActorNet and CriticNet at layer f, respectively, where s is the input state. The ActorNet calculates the policy π(a|s; λactor) and the CriticNet calculates the value function V(s; λcritic) for ℓ ∈ {1, 2,...} as follows:π(a|s;λactor) = fℓactor(...f1actor(s;λ1actor);λℓactor), andV(s;2critic) = wwhere fℓactor(xℓ;λℓactor) = ξℓactor(Wℓactorxℓ+ bℓactor) and fℓcritic(xℓ;λℓcritic) = ξξℓcritic(Wℓcriticxℓ+ bℓcritic) and xℓis some input vector for the ℓ-th layer. Note that x1= s and xℓ= fℓ-1(xℓ-1;λℓ-1)\). Here, ξactorandare the non-linear activation functions applied in the ℓ-th layer of ActorNet and CriticNet, respectively. The parameters of ActorNet are given by λactor= {Wℓactor, bℓactor|ℓ = 1, 2,...}, and the parameters of CriticNet are given by λcritic={Wℓcritic, bℓcritic|ℓ = 1, 2,...}.

[0123] To further details the whole procedure of PBPD described in Figures 6,7 and 8, the proposed PBPD is split into following steps.

[0124] Collect Trajectories: Run the policy π(a(ι) | s(ι);λ) and collect trajectories of (s(ι), a(ι), ρ(ι), s(ι + 1)) over multiple episodes.

[0125] Compute Returns: Calculate the returns ρ̂(ι) using the observed rewards, focusing on terminal and truncated states.

[0126] Estimate Advantages: Compute advantage estimates Â(ι) using temporal difference errors and value function estimates.

[0127] Compute Loss: Calculate the total loss for the PPO agent Lppothat consists of the loss of the policy network and the loss of the value network.

[0128] Update ActorNet & CriticNet: Use the clipped surrogate objective to perform policy updates, ensuring stable learning. In addition, minimize the loss of the value function to improve the accuracy of state value predictions.

[0129] By following these steps, PPO effectively trains ActorNet and CriticNet to handle the challenges posed by the reward structure in the MDP, ultimately leading to a policy that minimizes the loss, and thus the total energy consumption while satisfying QoS constraints. Each step is explained mathematically in the following.

[0130] Collect Trajectories: The collection of trajectories is performed by interacting the PPO agent with the mmWave mobile network environment. Through this interaction, the PPO agent sample different actions to explore the environment. Action selection is carried out by feeding the current state into the DNN. Given that the MDP involves a continuous action space, the generalized state-dependent exploration (gSDE) strategy is used to sample actions. In gSDE, actions are sampled from a normal distribution parameterized by (μ, σ), where g is derived from the DNN output: μ = πold(a(ι)|s(ι); λ). The variance of the action distribution a is learned over the course of training. This means that the variance is a parameter of the policy network and is adjusted during training to optimize the policy. This gSDE approach introduces state-dependent noise, allowing for a more effective exploration of the action space compared to constant noise strategies. By adapting the noise based on the current state, gSDE enhances coverage of the action space, aiding the policy in discovering optimal actions more efficiently. gSDE adapts more flexibly to changes in the environment, which is useful in the considered environments with dynamic state spaces.

[0131] Moreover, to improve exploration, the tanh function is applied to the action outputs sampled by the DNN. This squashing function maps the action space to the range [—1, 1], ensuring that actions remain within a bounded range. The tanh function is beneficial because it limits the magnitude of the actions, which helps in stabilizing the training process and prevents the policy from taking excessively large or unbounded actions that could destabilize learning. Further, the actions are normalized and mapped to the appropriate ranges for beam bandwidth partitioning and data allocation to ensure actions feasibility.

[0132] Upon sampling an action a(ι) from this normal distribution, the PPO agent calculates the log-probability of the action and evaluates the state value s(ι) using the critic network as will be discussed below. The environment then transitions to a new state s'(ι) based on UE mobility and CSI variations, and the reward function is applied to determine ρ(ι) referring to equation (17). The trajectory(s(ι), a(ι), ρ(ι), s'(ι)) is appended to a buffer, which is used to gather trajectories for training.

[0133] Compute Returns: The PPO agent computes the return ρ̂(ι) given by T-ιρ̂(ι) = Σγtρ(ι + t), (21)t=0where γ is the discount factor, and ρ(ι + t) is the reward received at time ι + t, defined in equation (17).

[0134] Estimate Advantages: First, the PPO agent computes the temporal difference error TD(ι) as TD(ι)= ρ(ι) + γ V(s(ι + 1);λcritic) - V(s(ι); λcritic). (22)

[0135] Next, the PPO agent computes the advantage Â(ι) that measures the discrepancy between the observed reward and the expected reward. Herein, the generalized advantage estimation (GAE) approach is adopted for a better learning. Using the GAE, the advantage estimate is given byÂ(ι) = ΣγtλgaetTD(ι + t), (23)t=owhere λgaeis a hyperparameter used to trade off bias versus variance for GAE.

[0136] Third, the agent calculates the probability ratio between the new policy and the old policy asπactor= π(a(ι)|s(ι); λactor)π(a(ι)|s(ι); λoldactor)

[0137] Compute Loss: A detailed and complete expression of the loss functions for both ActorNet and CriticNet, explicitly incorporating the reward (t) into the MDP is obtained as follows. First, the PPO agent is trained to minimize the CriticNet loss function given by the prediction error between the value function and the actual returns, which are calculated based on the return ρ̂(ι) in (21). That is, the value loss function for the CriticNet is the mean squared error (MSE) between the predicted value V(s(ι); λcritic) and the actual return ρ̂(ι) given asT—t X2‘LVF (^critic)=V(s(t); 2critic) - ykp(t + fc) )k=0 / where λcriticrepresents the CriticNet parameters. Next, the PPO agent trains the ActorNet to maximize the expected advantage while ensuring that the new policy does not deviate too much from the old policy by minimizing the ActorNet loss, acomputed using the clipped objective function and the ratio πactor;LCLIP(λactor) = −Et[min(πactorÂ(t), clip(πactor, 1 − ε, 1 + ε) Â(t))]. (26)

[0138] The combined loss function for PPO consists of the ActorNet loss, the CriticNet loss, and an entropy bonus to encourage exploration. It is expressed as LPPO= LCLIP(λactor) + c1LVF(λcritic) + c2H(π(·|s; λactor)), (27)where c1and c2are hyperparameters that balance the value function loss and the entropy loss, respectively. The Shannon entropy term is given byH(π(·|s; λactor)) = −Ea~π(·|s;λ)[log π(a|s; λactor)], (28)which encourages exploration by penalizing certainty in action selection.

[0139] Update ActorNet and CriticNet: Every futimesteps, PBPD performs an update where the old policy πold(·; λ) is updated to the new policy π(·; λ). The update process involves calculating the advantage estimates, Â(ι) of equation (23) and the surrogate objective function, LCLIP(λ), which depends on Â(ι), the old policy, the new policy, and includes a clipping term. The clipping term serves as a regularization technique to prevent excessive policy changes. The update process in PPO involves sampling minibatches from the trajectory buffer and applying multiple iterations of minibatch stochastic gradient descent (SGD) using backpropagation. It is worth mentioning that backpropagation faces different challenges in the present use casescenario of dynamic mobile networks, i) Sparse and Delayed Rewards: From equation (17), the rewards are sparse because the agent receives a reward of 0 at most time steps, and non-zero rewards only at terminal or truncated states. The delayed rewards make it challenging to attribute the effect of actions taken at earlier time steps to the final reward received, ii) Credit Assignment Problem: It is difficult to determine which actions were responsible for the eventual outcomes (terminal or truncated states), and such a problem is amplified by the sparse and delayed reward, iii) Variance in Reward: The reward received at terminal states can vary significantly, leading to high variance in the updates to the policy network, which can destabilize learning.

[0140] Despite these challenges, PPO can still effectively train ActorNet and CriticNet by using techniques designed to handle delayed and sparse rewards. As discussed previously, after collecting the trajectories by interacting with the environment using the current policy π(s(ι); λ), and computing the returns and advantages, the agent updates both the ActorNet and the CriticNet, respectively as follows. The ActorNet is updated by maximizing the clipped surrogate objective function LCLIP(λ). This involves performing several epochs of minibatch gradient descent:λactor← λactor− α̂actor∇λactorLCLIP(λactor)

[0141] The CriticNet is updated by minimizing the loss of the value function LVF(Acritlc), also using the minibatch gradient descent:λcritic← λcritic− α̂critic∇λcriticLVF(λcritic)where α̂actorand α̂criticare the learning rate for the ActorNet and CriticNet, respectively and VF is a Value Function.

[0142] Over many iterations of collecting data, computing rewards, and updating the networks, the policy converges to an optimal or near-optimal strategy. This policy found by the PPO agent in the proposed MDP context refers to the strategy that the agent learns to optimally allocate beams and power in the wireless communication system. Once the training converged, the optimized policy π(a|s; λ) can be inferred from the ActorNet. Given a new state s(t), ActorNet outputs the optimal mean and standard deviation (std) of the action a. Using the trained mean and std, the optimal action is sampled. Clearly, this process is influenced by s(t), but thanks to offline training and the techniques used in the proposed PBPD algorithm,the PPO agent can adapt quickly to the new unobserved state. The optimized policy maximizes the expected cumulative reward while ensuring guaranteed QoS requirements.

[0143] Simulation Results

[0144] Benchmarking

[0145] In this section, it is proposed to use hybrid online-offline and iterative-DRL solutions for the BPO problem, serving as conventional baselines for comparison with the proposed DRL approach (PBPD). Two conventional methods are introduced: (i) GBCD, which uses two greedy strategies, and (ii) GBOP, which combines greedy and optimal strategies. Both methods address beam bandwidth partitioning cr(t) and power allocation p(t) for all UEs at each slot t. GBCD derives p(t) via the NR method by optimizing data allocation y(t) as detailed in Lemma 1, while GBOP computes p(t) directly using convex optimization. To assess and validate the superior performance of PBPD, it is further compared to Greedy Bandwidth and PPO for Power (GBPP) and Greedy Bandwidth and PPO for Data (GBPD). In GBPP, the online BO phase is proposed for bandwidth partitioning and PPO is proposed for PO. In contrast, in GBPD, the online BO phase is proposed for bandwidth partitioning and PPO is proposed for DO.

[0146] GBCD: The fully greedy approach GBCD has two phases. The first phase is online BO to find cr(t), and the second phase is offline DO to determine y(t). The BO phase primarily relies on a proportional fair subcarrier allocation, where each UE n belonging to traffic class x(ri) is assigned a weight wn(t) at each slot t as follows:z.x >enQn(t)WnW- ||Hn(t)lMt)’where enis the priority level of class / (n), Qn(t) is the current remaining data for UE n (initially Qn(t) = Qn), tn(t) is the history of subcarrier allocation for UE n and increases as tn(t) <- tn(t) + Kn(t), and ||Hn(t)|| is the CSI norm. The sum of all weights in the slot t is denoted by W(t)) = £nwn(t). The pseudo-code of GBCD is shown in Figure 9.

[0147] GBOP: The mixed greedy-optimal algorithm, GBOP, also consists of two phases. The first phase is identical to that in GBCD, whereas the second phase involves determining the optimal power allocation directly through convexoptimization (offline PO). To apply convex optimization, after obtaining cr(t) using the BPO phase, problem (Pl) is transformed into a convex optimization problem as follows:T Nminimize / / En(t), (P2)t=l n=l subject to pn(t) ∈ [0, Pmax], ∀ n ∈ N, t ∈ T, (P2a)Σpn(t) ∈ {0, 1}, ∀t ∈ T, (P2b)−Σ Bsτ lg(1 + pn(t)Λk,n(t)) ≤ −Qn(1 − qχ(n)), ∀ n ∈ N. (P2c)t=i k=l

[0148] Since the logarithmic function is concave, the constraints in (P2c) are convex and can be solved using a convex solver. The pseudo-code of GBOP is shown in Figure 10.

[0149] Worst-Case Complexity Analysis of Iterative Solutions

[0150] The BO phase of GBCD clearly requires O(KNT) worst-case complexity. The second phase, however, is dominated by the worst-case complexity of the NR method, which is given by O( / GV / max+ N In A). Here, / maxis the maximum number of iterations set for the NR method. Thus, the worst-case complexity of GBCD is O(KNTImax+ NT ln N). The worst-case complexity of GBOP is dominated by the complexity of solving (P2).

[0151] Results and Discussions

[0152] Simulation Setup

[0153] In the simulation, a highway is modeled as a rectangular area of size 24m-by-500m. The BS is located at the coordinates (250, —20). The width of the highway is divided into six lanes. Each lane has a width of 4 m. Three lanes are reserved for right-to-left direction and the remaining lanes are reserved for left-to-right direction. UEs are modeled as vehicles moving on the highway. The vehicles in a lane move at the same speed and the possible speed of each vehicle is taken from the set {60, 80, 100} km / h. Vehicles are generated according to a spatial Poisson process and the average distance between vehicles (in the same lane) is set to 2.5s x v, where v is the vehicle speed in a lane. Monte Carlo simulations are performed, and the results are averaged over 500 random realizations. Unless otherwise specified, theimportant simulations parameters are given in Table I.Settings and notations ValueNumber of UEs, N 5Number of intervals, T 100Number of BS antennas, M 64Number of subcarriers, K 504Carrier frequency, fc28 GHzSubcarrier spacing, Bsc120 kHzSlot duration, r 1 msSimulation period 100 msMaximum BS transmit power, Pmax 47 dBmNoise power, No -174 dBm / HzTABLE I: Network parameters and simulation settings

[0154] Four traffic classes and four scenarios are considered, where the data requirements of the four classes for the ith scenario are denoted as GBRi. In each scenario, the data requirements for the traffic classes change, and UEs are randomly assigned to these classes. This setup allows to evaluate the system's adaptability and efficiency across different traffic conditions.

[0155] PPO Parameters

[0156] The PPO agent is implemented in Pytorch and Gymnasium using stable-baselines3. The PPO agent uses the actor-critic architecture, where ActorNet and CriticNet are modeled as sequential containers with two hidden linear layers each with Tanh nonlinear activation functions and 400 and 300 neurons, respectively. The simulation parameters for PPO are given in Table II and the PPO hyperparameters are fine-tuned using grid search (for PBPD, GBPP, and GBPD).Parameters and Hyperparameters ValueNumber of timesteps, E 5 x 106timestepsActorNet learning rate, α̂actorLinearly scheduled (initially 10-4) CriticNet learning rate, β̂criticLinearly scheduled (initially 10-4) Optimizer AdamAction std Initially 1 (learnable parameters) Exploration strategy gSDEgSDE sample frequency 16Discount factor for reward, λ 0.99Clipping parameter, e 0.2Value loss coefficient, cl 0.5Entropy coefficient, c2 10-4Size of mini-batch 128Number of epochs 20GAE coefficient, λgae0.95PPO update frequency, fu4096 timestepsHidden layers of ActorNet [400,300]Hidden layers of CriticNet [400,300]Activation functions, ^actorand ^C"1IC_ Tanh & Tanh _TABLE II: PPO parameters and learning settings

[0157] The following PPO-based results are achieved in the deployment scenario, after being trained only once with specific network parameters, demonstrating the effectiveness of the method without requiring further training.Mean Std Dev Min Max Median GBPP 0.017 0.001 0.012 0.019 0.017 GBPD 0.113 0.042 0.012 0.304 0.108 PBPD 0.161 0.070 0.063 0.445 0.152 Table III: Reward Convergence Statistics for the PPO-based algorithms

[0158] In Table III, the reward convergence statistics of different PPO-based algorithms are shown. Among GBPP, GBPD, and PBPD, the latter stands out as the superior choice based on its performance metrics. PBPD has the highest mean performance (0.16), indicating that it consistently delivers better average results. Despite its variability (Std Dev = 0.07), PBPD's potential to achieve significantly higher maximum rewards (up to 0.45) makes it a powerful candidate for applications where achieving top reward is critical. This performance gain is attributed not only to improved power allocation, but also to effective data allocation, which plays a crucial role in optimizing the overall system performance. Note that the reward is always positive since the PPO agent never reached a truncated state due to the time length and the moderate data requirements. GBPP demonstrates the lowest reward convergence despite being more stable, reflecting the challenges associated with optimizing power directly. The superior performance of PBPD underscores the limitations of combining greedy strategies with PPO, emphasizing the benefits of a fully PPO-based approach that optimizes both bandwidth and data allocation.

[0159] The results in Figure 11 demonstrate the effectiveness of bandwidth partitioning and data allocation using the fully PPO algorithm in enhancing energy efficiency compared to traditional power allocation methods. The GBPP method shows a significant improvement in energy efficiency as the required data increases, compared to GBCD and GBPD, which utilize different resource allocation strategies. The GBOP method outperforms the PBPD method since it leverages an optimal algorithm using convex optimization for power allocation. Notably, the PBPD method outperforms all other benchmarks, including GBPP, across all GBR levels. This indicates that allocating data rather than power yields better energy efficiency. This finding underscores the potential of data allocation in optimizing the performance ofmobile communication systems, especially in dynamic scenarios demanding varying data requirements.

[0160] In Figure 12, the GBPP method demonstrates the highest power consumption across all GBR levels. The GBOP method, which employs optimal power allocation using convex optimization, achieves the lowest power consumption but is more complex. The PBPD method shows competitive power consumption close to GBOP with significantly lower complexity, making it the preferred choice. This is due to the use of PPO for both bandwidth and data allocation, whereas direct optimization of power in GBPP leads to higher power consumption due to the hard exploration in the policy optimization. Intermediate power consumption levels are observed for the GBCD method and the GBPD method. This figure highlights the trade-offs between different resource allocation strategies in terms of power usage and complexity.

[0161] Figure 13 demonstrates that the PBPD algorithm achieves the lowest average percentage of slots used in all GBR values, highlighting its efficiency in minimizing resource utilization. The lower slot usage observed in PBPD indicates its potential to turn off BS during periods of inactivity, contributing to significant energy savings. This is important for energy-efficient network design, as is the case of the network referred to herein, particularly in scenarios with intermittent traffic.Furthermore, by utilizing fewer slots, PBPD can allocate higher data rates at those active slots, efficiently optimizing energy / rate tradeoff during data bursts or periods of high demand. The fully PPO-leaming approach (PBPD) effectively optimizes resource allocation, achieving the best balance between meeting required data transmission and minimizing slot utilization. This highlights its superiority in resource-sensitive and energy-efficient communication network strategies. In contrast, other algorithms exhibit higher slot utilization, suggesting that they are less energy efficient.

[0162] Figure 14 illustrates the cumulative distribution function (CDF) of energy efficiency (Mb / J) for the traffic demands of GBR4, which corresponds to the average data requirements of 9.69 Kb. The PBPD approach demonstrates superior performance, using PPO for both bandwidth and data. It consistently outperforms other benchmarks. This makes PBPD a robust and practical solution for various deployment scenarios.

[0163] Figure 15 illustrates the CDF of power consumption (watts) for thetraffic demands of GBR4 in various solutions. The PBPD approach demonstrates superior performance, using PPO for both bandwidth and data. It consistently outperforms other benchmarks such as GBPP, GBCD, GBPD, and the more complex GBOP. Notably, PBPD shows significantly lower power consumption across the entire GBR range, highlighting the efficacy of integrating PPO for both bandwidth and data. This makes PBPD a robust and practical solution for various deployment scenarios.

[0164] Turning to Figure 16, there is provided a computer implemented method 1600 for executing a proximal policy optimization (PPO) agent for optimizing bandwidth and power allocated to beams established between moving mobile devices (MDs) and a base station (BS), to improve energy efficiency. The method comprises determining, step 1601, using the PPO agent, the bandwidth to be allocated to each of the MDs, by predicting data to be transmitted for each MD. The method comprises, step 1602, allocating at least one subcarrier to each MD for a time slot. The method comprises, step 1603, computing the power allocation for each beam, established between the MD and the BS for the time slot, based on the corresponding determined bandwidth and allocated at least one subcarrier.

[0165] The system described herein is mainly directed towards downlink transmissions. However, a person skilled in the art would understand that it can be extended to also cover uplink transmissions. In such an uplink scenario, a PPO agent is also deployed on the MD side and the system is a multi -agent system.

[0166] The data to be transmitted for each MD may be predicted based on: channel state information (CSI), the position of the MD, the distance between the MD and the BS and a delay and a phase shift of each antenna to have frequency dependent beams pointing at different directions.

[0167] The PPO agent may be trained, at each time slot, using a function that rewards minimizing total energy consumption.

[0168] There may be two modes of operation of the system, an offline mode (training) and an online mode (deployment). During training, the agent may operate in a simulated environment. During deployment, the well-trained model is used in an online fashion at each timeslot.

[0169] The PPO agent may be trained by collecting trajectories, and a policy of the PPO agent may be updated based on the trajectories.

[0170] The trajectories may comprise {state, action, next state, reward} overmultiple episodes. The policy may be implemented through a deep neural network (DNN) comprising an actor that outputs an action.

[0171] The bandwidth for the time slot may be allocated using a plurality of subcarriers that are either contiguous or non-contiguous.

[0172] Computing the power allocation for each beam may be done by applying a Newton-Raphson (NR) method; by first applying data allocation and then computing the power through Newton-Raphson (NR) method.

[0173] Upon detecting that a total allocated power is above a total power available at the BS, the power allocation for one of more beams may be reduced.

[0174] The bandwidth allocation may be recomputed for the one of more beams based on the reduced power allocation.

[0175] It should be noted that methods and steps described herein are, generally, computer implemented methods and steps. The term computer may be interpreted as having different meanings, such as explained next, for example.

[0176] Referring to Figure 17, there is provided a network node (HW) 1701, in which functions and steps described herein can be implemented.

[0177] The network node 1701 may be a server, a network node, a radio base station, or any other computing device which may be part of a cloud computing system, edge computing system, or which may be a standalone device.

[0178] The network node, and / or the mobile device, may be configured and used for executing the proximal policy optimization (PPO) agent described herein for optimizing bandwidth and power allocated to beams established between moving mobile devices (MDs) and a base station (BS). Executing the agent(s) as the technical effect of improving energy efficiency of the system.

[0179] The network node 1701 comprises processing circuitry 1703 and memory 1705. The memory 1705 can contain instructions executable by the processing circuitry 1703 whereby functions and steps described herein may be executed to provide any of the relevant features and benefits disclosed herein.

[0180] The network node 1701 may also include non-transitory, persistent, machine-readable storage media 1707 having stored therein software and / or instruction 1709 executable by the processing circuitry 1703 to execute functions and steps described herein. The network node may also include network interface(s) and a power source.

[0181] The instructions 1709 may include a computer program for configuringthe processing circuitry 1703. The computer program may be stored in a physical memory local to the device, which can be removable, or it could alternatively, or in part, be stored in the cloud. The computer program may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0182] Referring to Figure 18, there is provided a virtualization environment 1800 in which functions and steps described herein can be implemented.

[0183] The virtualization environment 1800 (which may go beyond what is illustrated in Figure 18), may comprise systems, networks, servers, nodes, devices, etc., that are in communication with each other either through wire or wirelessly, e.g. through a network interface component (NIC) comprising physical network interface(s). Some or all of the functions and steps described herein may be implemented as one or more virtual components (e.g., via one or more applications, components, functions, virtual machines, containers, etc.) executing on one or more physical apparatus in one or more networks, systems, environment, etc.

[0184] A virtualization environment provides hardware 1801 comprising processing circuitry 1803 and memory 1805. The memory 1805 can contain instructions executable by the processing circuitry 1803 whereby functions and steps described herein may be executed to provide any of the relevant features and benefits disclosed herein.

[0185] The hardware 1801 may also include non-transitory, persistent, machine-readable storage media 1807 having stored therein software and / or instruction 1809 executable by the processing circuitry 1803 to execute functions and steps described herein.

[0186] The instructions 1809 may include a computer program for configuring the processing circuitry 1803. The computer program may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program may be stored in a physical memory local to the hardware 1801, which can be removable, or it could alternatively, or in part, be stored in the cloud. The computer program may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

[0187] Referring again to Figures 17 and 18, there is provided a network node 1701, 1801 operative to execute a proximal policy optimization (PPO) agent foroptimizing bandwidth and power allocated to beams established between moving mobile devices (MDs) and a base station (BS), to improve energy efficiency. The network node comprises processing circuits or circuitry 1703, 1803 and a memory 1705, 1805. The memory contains instructions executable by the processing circuits whereby the network node is operative to determine, using the PPO agent, the bandwidth to be allocated to each of the MDs, by predicting data to be transmitted for each MD. The network node is operative to allocate at least one subcarrier to each MD for a time slot. The network node is operative to compute the power allocation for each beam, established between the MD and the BS for the time slot, based on the corresponding determined bandwidth and allocated at least one subcarrier.

[0188] The data to be transmitted for each MD may be predicted based on: channel state information (CSI), the position of the MD, the distance between the MD and the BS and a delay and a phase shift of each antenna to have frequency dependent beams pointing at different directions.

[0189] The PPO agent may be trained, at each time slot, using a function that rewards minimizing total energy consumption.

[0190] The PPO agent may be trained by collecting trajectories, and a policy of the PPO agent may be updated based on the trajectories. The trajectories may comprise {state, action, next state, reward} over multiple episodes.

[0191] The policy may be implemented through a deep neural network (DNN) comprising an actor that outputs an action.

[0192] The bandwidth for the time slot may be allocated using a plurality of subcarriers that are either contiguous or non-contiguous.

[0193] Computing the power allocation for each beam may be done by applying a Newton-Raphson (NR) method.

[0194] Upon detecting that a total allocated power is above a total power available at the BS, the power allocation for one of more beams may be reduced.

[0195] The bandwidth allocation may be recomputed for the one of more beams based on the reduced power allocation.

[0196] The network node is further operative to execute any of the steps described herein.

[0197] Referring again to Figures 17 and 18, there is provided a non-transitory computer readable media 1707, 1807 having stored thereon instructions 1709, 1809 for executing a proximal policy optimization (PPO) agent for optimizing bandwidthand power allocated to beams established between moving mobile devices (MDs) and a base station (BS), to improve energy efficiency. The instructions comprise determining, using the PPO agent, the bandwidth to be allocated to each of the MDs, by predicting data to be transmitted for each MD. The instructions comprise allocating at least one subcarrier to each MD for a time slot. The instructions comprise computing the power allocation for each beam, established between the MD and the BS for the time slot, based on the corresponding determined bandwidth and allocated at least one subcarrier.

[0198] The instructions may comprise further instructions for executing any of the steps described herein.

[0199] Modifications will come to mind to one skilled in the art having the benefit of the teachings presented in the foregoing description and the associated drawings. Therefore, it is to be understood that modifications, such as specific forms other than those described above, are intended to be included within the scope of this disclosure. The previous description is merely illustrative and should not be considered restrictive in any way. The scope sought is given by the appended claims, rather than the preceding description, and all variations and equivalents that fall within the range of the claims are intended to be embraced therein. Although specific terms may be employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

CLAIMS1. A computer implemented method for executing a proximal policy optimization (PPO) agent for optimizing bandwidth and power allocated to beams established between moving mobile devices (MDs) and a base station (BS), to improve energy efficiency, the method comprising:determining, using the PPO agent, the bandwidth to be allocated to each of the MDs, by predicting data to be transmitted for each MD;allocating at least one subcarrier to each MD for a time slot; and computing the power allocation for each beam, established between the MD and the BS for the time slot, based on the corresponding determined bandwidth and allocated at least one subcarrier.

2. The method of claim 1, wherein data to be transmitted for each MD is predicted based on: channel state information (CSI), the position of the MD, the distance between the MD and the BS and a delay and a phase shift of each antenna to have frequency dependent beams pointing at different directions.

3. The method of claim 1 or 2, wherein the PPO agent is trained, at each time slot, using a function that rewards minimizing total energy consumption.

4. The method of any one of claims 1 to 3, wherein the PPO agent is trained by collecting trajectories, and wherein a policy of the PPO agent is updated based on the trajectories.

5. The method of claim 4, wherein the trajectories comprise {state, action, next state, reward} over multiple episodes.

6. The method of claim 4, wherein the policy is implemented through a deep neural network (DNN) comprising an actor that outputs an action.

7. The method of any one of claims 1 to 6, wherein the bandwidth for the time slot is allocated using a plurality of subcarriers that are either contiguous or noncontiguous.

8. The method of any one of claims 1 to 7, wherein computing the power allocation for each beam is done by applying a Newton-Raphson (NR) method.

9. The method of any one of claims 1 to 8, wherein, upon detecting that a total allocated power is above a total power available at the BS, the power allocation for one of more beams is reduced.

10. The method of claim 9, wherein bandwidth allocation is recomputed for the one of more beams based on the reduced power allocation.

11. A network node operative to execute a proximal policy optimization (PPO) agent for optimizing bandwidth and power allocated to beams established between moving mobile devices (MDs) and a base station (BS), to improve energy efficiency, the network node comprising processing circuits and a memory, the memory containing instructions executable by the processing circuits whereby the network node is operative to:determine, using the PPO agent, the bandwidth to be allocated to each of the MDs, by predicting data to be transmitted for each MD;allocate at least one subcarrier to each MD for a time slot; andcompute the power allocation for each beam, established between the MD and the BS for the time slot, based on the corresponding determined bandwidth and allocated at least one subcarrier.

12. The network node of claim 11, wherein data to be transmitted for each MD is predicted based on: channel state information (CSI), the position of the MD, the distance between the MD and the BS and a delay and a phase shift of each antenna to have frequency dependent beams pointing at different directions.

13. The network node of claim 11 or 12, wherein the PPO agent is trained, at each time slot, using a function that rewards minimizing total energy consumption.

14. The network node of any one of claims 11 to 13, wherein the PPO agent is trained by collecting trajectories, and wherein a policy of the PPO agent is updated based on the trajectories.

15. The network node of claim 14, wherein the trajectories comprise {state, action, next state, reward} over multiple episodes.

16. The network node of claim 14, wherein the policy is implemented through a deep neural network (DNN) comprising an actor that outputs an action.

17. The network node of any one of claims 11 to 16, wherein the bandwidth for the time slot is allocated using a plurality of subcarriers that are either contiguous or non-contiguous.

18. The network node of any one of claims 11 to 17, wherein computing the power allocation for each beam is done by applying a Newton-Raphson (NR) method.

19. The network node of any one of claims 11 to 18, wherein, upon detecting that a total allocated power is above a total power available at the BS, the power allocation for one of more beams is reduced.

20. The network node of claim 19, wherein bandwidth allocation is recomputed for the one of more beams based on the reduced power allocation.

21. A non-transitory computer readable media having stored thereon instructions for executing a proximal policy optimization (PPO) agent for optimizing bandwidth and power allocated to beams established between moving mobile devices (MDs) and a base station (BS), to improve energy efficiency, the instructions comprising:determining, using the PPO agent, the bandwidth to be allocated to each of the MDs, by predicting data to be transmitted for each MD;allocating at least one subcarrier to each MD for a time slot; and computing the power allocation for each beam, established between the MD and the BS for the time slot, based on the corresponding determined bandwidth and allocated at least one subcarrier.

Images