Backup and / or restore controller and method in a leaderless database system

Abstract

A method for handling de-duplication in a leaderless distributed database system is provided. The database system comprises a set of datacenters. No datacenter in the set of datacenters is configured as a dedicated backup datacenter. A datacenter is selected (902) from the set of datacenters in the distributed database system for a keyspace associated with a backup or restore operation. A replication factor for the keyspace is selected (903). Based on a replication factor for the keyspace, the backup and / or restore operation is assigned (904) to one or more nodes in the selected datacenter. The backup and / or restore operation is performed (905).

Description

[0001] BACKUP AND / OR RESTORE CONTROLLER AND METHOD IN A LEADERLESS DATABASE SYSTEM

[0002] TECHNICAL FIELD

[0003] Embodiments herein relate to a backup and or restore controller and method therein. In some aspects, they relate to handling de-duplication in a leaderless database system.

[0004] BACKGROUND

[0005] In a typical wireless communication network, wireless devices, also known as wireless communication devices, mobile stations, stations (STA) and / or User Equipment (UE), communicate via a Wide Area Network or a Local Area Network such as a Wi-Fi network or a cellular network comprising a Radio Access Network (RAN) part and a Core Network (CN) part. The RAN covers a geographical area which is divided into service areas or cell areas, which may also be referred to as a beam or a beam group, with each service area or cell area being served by a radio network node such as a radio access node e.g., a Wi-Fi access point, a Base Station (BS) or a radio base station (RBS), which in some networks may also be denoted, for example, a Base Station (BS), a NodeB, eNodeB (eNB), or gNodeB (gNB) as denoted in Fifth Generation (5G) telecommunications. A service area or cell area is a geographical area where radio coverage is provided by the radio network node. The radio network node communicates over an air interface operating on a radio frequency with the wireless devices within the range of the radio network node.

[0006] 3rd Generation Partnership Project (3GPP) is the standardization body for specifying the standards for the cellular system evolution, e.g., including 3G, 4G, 5G and the future evolutions. Specifications for Evolved Universal Terrestrial Radio Access (E-UTRA) and Evolved Packet System (EPS) have been completed within the 3GPP. In 4G also called a Fourth Generation (4G) network, EPS is core network and E-UTRA is radio access network. In 5G, 5GC is core network, NR is radio access network. As a continued network evolution, the new release of 3GPP specifies a 5G network also referred to as 5G New Radio (NR) and 5G Core (5GC).

[0007] Frequency bands for 5G NR are being separated into two different frequency ranges, Frequency Range 1 (FR1) and Frequency Range 2 (FR2). FR1 comprises sub-6GHz frequency bands. Some of these bands are bands traditionally used by legacy standards but have been extended to cover potential new spectrum offerings from 410 MHz to 7125 MHz. FR2 comprises frequency bands from 24.25 GHz to 52.6 GHz. Bands in this millimeter wave range have shorter range but higher available bandwidth than bands in the FR1.

[0008] Multi-antenna techniques may significantly increase the data rates and reliability of a wireless communication system. For a wireless connection between a single user, such as UE, and a base station (BS), the performance is in particular improved if both the transmitter and the receiver are equipped with multiple antennas, which results in a Multiple-Input Multiple-Output (MIMO) communication channel. This may be referred to as Single-User (SU)-MIMO. In the scenario where MIMO techniques is used for the wireless connection between multiple users and the base station, MIMO enables the users to communicate with the base station simultaneously using the same time-frequency resources by spatially separating the users, which increases further the cell capacity. This may be referred to as Multi-User (MU)-MIMO. Note that MU-MIMO may benefit when each UE only has one antenna. The cell capacity can be increased linearly with respect to the number of antennas at the BS side. Due to that, more and more antennas are employed in BS. Such systems and / or related techniques are commonly referred to as massive MIMO.

[0009] The telecommunication industry processes distributed data and this distribution is directly proportional to the size of the market a telecommunication company attends. Another factor that affects data distribution are the different kinds of data that telco systems need to manage, as example:

[0010] • Customer data: who are the subscribers?

[0011] • Offerings and pricing: what are the services the operator offer to the subscribers and how much these need to be charged?

[0012] • Calls, data traffic and sms data: what are the subscribers doing?

[0013] On top of these two factors, for economical, data protection, robustness, or even legal aspects, data needs to be stored with replication, both locally and across physically distributed locations (sites) and kept for long periods of time.The Apache Cassandra database is used as an example of a distributed database system and target system. As preparation, follows discussions on basic concepts in Cassandra.

[0014] Cassandra Concepts

[0015] The simplest unit of computation in Cassandra is a node. Each database node holds part of the data that a datacenter stores, therefore a datacenter is a group of database nodes. Within one datacenter the nodes can be divided into a set of logical racks to evenly distribute the data between the racks within the datacenter, which offers yet another level of fault tolerance and high availability. Further, the data sharing among the nodes in a datacenter is done whenever a new node joins the datacenter, to achieve horizontal scaling within the datacenter. A collection of datacenters is called a Cassandra cluster.

[0016] In Figure 1, “CIL1” and “CIL2” are two datacenters in a Cassandra cluster, with one rack each. The rack in each of the datacenters hosts 3 Cassandra nodes.

[0017] Cassandra storage engine is based on several components, one being the Sorted String Tables (SSTables) - which are immutable files which are written to disk once Cassandra has accumulated a pre-defined data-set size in memory. Since Cassandra is a leaderless distributed database, these SSTables might look very different between the Cassandra nodes in the same cluster. These very same SSTables are on a node-by-node basis compacted to create a new immutable file (SSTable) to free up storage on disk.

[0018] To achieve data replication, datacenters need to keep more than one single copy of a given piece of data, so that in case of partial system unavailability, another copy of the data is still reachable. A general recommendation is to use a replication factor of 3 in a distributed database, which means that each piece of data will be stored in three different database nodes.

[0019] Cassandra stores data in tables, whose schema defines the columns, their respective data type and other properties of the table. Tables are encapsulated in keyspaces that describe the replication strategy of all underlying tables and their data. Replication strategy outlines the datacenters and number of copies of data that should be stored in the corresponding datacenter. Assuming a replication factor of 3, the data in alltables of a given keyspace is stored in 3 separate nodes within all the datacenters included in the replication strategy.

[0020] For illustration, in Figure 2, in a Cassandra Cluster with 4 datacenters, given a keyspace “customer” with two tables “offerings” and “pricing” using a replication strategy of replicating to datacenters “CIL1” and "CIL3” with replication factor of 3, will ensure 3 copies of data in “offering” and “pricing” tables to be stored in “CIL1” and “CIL3” each. From the example, each entry in the “offering” and “pricing” table is stored 6 times across the complete Cassandra Cluster. In the event of node failure in any given datacenter, the data can be retrieved from another available node holding one copy of the data.

[0021] Within one datacenter we can divide the nodes into a set of logical racks to evenly distribute the data between the racks within the datacenter. Racks are logical grouping of nodes to ensure the data is evenly spread in the datacenter and avoid redundant copies of data in the same rack. When many racks exist in a datacenter, Cassandra replicates data per rack. For illustration, in Figure 3, given each datacenter has 3 racks, and each rack has 3 nodes, one copy of data in keyspace “customer” will be stored on each rack of datacenters “CIL1” and “CIL3”. Therefore, each rack will hold one copy of data in one of the 3 nodes in the rack and 6 copies of data will exist across the complete Cassandra cluster. If one rack were to be unavailable, the data can be read from another node from a different rack from the datacenter.

[0022] Another concept, which is not really a Cassandra concept, but is usually adapted to distributed databases is the site concept. A site refers to a physical location, for example Karlskrona, Sweden. In one site, one or more datacenters can be deployed to store part of the application data. An operator can choose to have geographically redundancy on this data, by deploying datacenters on yet another psychically separate site, for example Stockholm, Sweden.

[0023] Using replication factor equals to 3 ensures high availability and good consistency, so that one of the nodes that holds a given piece of data can be down for whatever reason. However, this has its drawbacks when it comes to performing a backup on the distributed system, as this will add significant cold storage costs. This is because the same piece of data will now be backup up 3 times per datacenter where the data is stored.In a more complex system, where multiple regions and sites are deployed, this could also mean that the same piece of data is potentially backed up also in different sites. To illustrate, the following example depicts a system spread in three sites with global data, i.e. data that must be available in all the sites, partitioned data, i.e. data that is local in parts of the system and in this case local to the site. In a standard implementation of the backup / restore solution, a backup will be taken on all 18 database nodes depicted in Figure 4, according to the data distribution from Table 1.

[0024]

[0025] Table 1

[0026] The system backs up 9 replicas of the global data (replication factor 3, stored in three datacenters, one per site) and 6 replicas of the data from each of the three partitions (replication factor 3, stored in two different datacenters, in different sites for geographical redundancy).

[0027] Assuming each database node has a data disk size of 360 GiB, the data we store for the backup of global and partitioned data will be 3240 GiB and 6480 GiB respectively. This will be our boiler plate implementation of a backup and restore solution, that all the data in the system is fully backed up and stored in a cold storage (for example Amazon S3). When emergency strikes, all data must be transferred back to the live system using a restore operation.

[0028] To mitigate the cost of having duplicated data stored in the cold storage, a general recommendation is to have one dedicated backup / restore datacenter, where only one copy of the live data is replicated to a so-called de-duplication datacenter. To configure this recommendation, it is enough to configure data replication as in Table 2.

[0029] Table 2: Data distribution for base example system with dedicated backup / restore datacenter

[0030]

[0031] Table 2

[0032] Figure 5 brings an illustration on how a dedicated backup restore datacenter is added in each site. Meaning that all data actively being written to Site 1 datacenters, will be passively copied to ’’Backup CILx” datacenter using a replication factor of 1, likewise for the other two sites.

[0033] SUMMARY

[0034] As a part of developing embodiments herein a problem was identified by the inventors and will first be discussed.

[0035] Maintaining data integrity and availability is critical in distributed databases such as Apache Cassandra. This involves comprehensive backup and restore methods, particularly for recovery from various scenarios like as catastrophic failures that impose the reinstallation of the entire cluster, data corruption, accidentally deletions and other similar circumstances.

[0036] The native solution that comes in Cassandra where backups are taken as snapshots with the help of a native tool called nodetool. The “nodetool snapshot” command is employed to create consistent snapshots of all keyspaces or specific keyspaces. This operation generates hard links to the SSTables, ensuring a consistent backup without downtime.

[0037] There are challenges with the native solution because the steps needed to do backup and restore are manual. It involves a number of arduous activities, including running the nodetool snapshot command and copying the snapshot files to backup storage or any cloud storage provider, such as AWS's S3. This is extremely cumbersome for large-scale deployments. Since backup and restore operations are performed manually, the time taken to complete them can be lengthy, resulting in longer recovery times. Additionally, since snapshots are taken at the node level, they can causeconsistency issues across the cluster because they are taken at different intervals and the solution doesn't provide any coordination between the snapshots taken.

[0038] One related invention is Medusa https: / / github.com / thelastpickle / cassandra-medusa, an open-source project developed and maintained by the Cassandra community that is explicitly designed to provide robust backup and restore capabilities for Apache Cassandra databases. It is not included in the core Cassandra release, but it is frequently used and supported by the community to overcome the limitations of native backup and restoration methods.

[0039] Medusa offers automated backups, which ensures consistent backup performance without requiring human intervention by taking backups at scheduled times. It supports integration with several cloud storage providers, including Minio and AWS S3, making scalable and effective backup data storage solutions possible. It minimizes the impact on the Cassandra nodes' performance, optimizes backup and restore procedures, and offers consistency.

[0040] Although Medusa has many benefits, it also has drawbacks. In a container-based deployment, Medusa is deployed as a side car container with Cassandra where side car containers are secondary containers that run along with the main container within the same pod. A side car requires fixed resources allocated at the time of deployment for each pod to ensure quality of service. Given backups are scheduled operations, the resources allocated to the Medusa side car will not be utilized outside the scheduled backup cycle. In a complex system where multiple regions and sites are deployed, the cost of resources required to deploy Medusa will be considerably high. Additionally, when performing a backup, Medusa stores the data from each Cassandra node in the cluster thereby storing as many copies of data as defined by the replication factor across the Cassandra cluster which incurs significant cold storage cost.

[0041] One of the biggest drawbacks is that the amount of data restored onto a cluster will be the size of the tables from the node backup multiplied by the replication factor, meaning there is no de-duplication of the data before its being restored. For example, say that there is a backup table with size 20Gi stored in a cloud storage provider say Amazon’s S3. To restore it, Medusa downloads the table from Amazon S3, stores it in a temporary folder and streams it to the Cassandra nodes. The table is streamed to all the other nodes based on the replication strategy (datacenters and respective replication factor) configured before the backup was taken. While streaming, the files in the temporary directory cannot be deleted until all the data has been streamed across all nodes. This causes a huge utilization of resources where the storage required in this casewould be 20Gi * replication factor * Number of datacenters the data is replicated to. If we consider the replication factor to be 3 for replication on two datacenters, then in total the memory required would be the current memory resources plus an additional 20Gi * 3 * 2 = 120GL Therefore, the storage limit of each Cassandra disk needs to be scaled exponentially depending on the expected restore data size.

[0042] Another drawback is that the restore times are significantly longer. One reason is that the downloads and streaming of tables are done in a sequential way. For example, if there are 5 backup tables for restore, Medusa downloads all the tables before starting to stream the data. If there are network delays, then it contributes to longer restore times.

[0043] Another related invention is the DataStax OpsCenter which works similar to Medusa and therefore has similar drawbacks.

[0044] The state of the art currently involves either adding a dedicated datacenter with replication factor 1 , meaning only 1 copy of the data exists to the already-existing Cassandra Cluster to take backups and prevent data duplication or using a post processing client which processes the backup data to de-duplicate it at the partition level.

[0045] Data deduplication is an essential step before making a backup. If not done, the result will be a massive backup file that will require more storage space and longer backup / restore durations. Data deduplication by post-processing clients is resource intensive and time-consuming since the backup contains the data from all nodes which need to be read and de-duplicated leading to longer backup times.

[0046] With companies requiring reduced Total Cost of Ownership (TCO) to have a good profitability, for large-scale deployments taking huge backups and restoring the data back on to the Cassandra nodes will consume additional resources not only memory but also CPU and network bandwidth, which might also impact the performance of the Cassandra nodes as well. Therefore, in addition to the benefits that Medusa provides, a better solution is needed to reduce TCO as well as time taken for restore operations.

[0047] Semantic de-duplication

[0048] Another approach separated from having a dedicated datacenter to handle the deduplication layer would be to use the “sematic de-duplication" approach.

[0049] Semantic de-duplication tracks changes in data across Cassandra cluster and one copy of each change is streamed to archival storage periodically. This archival storage can be used to sync and / or restore data to one or more Cassandra clusters. Or alternatively, take a snapshot of the data in archival storage to be stored in a cold storage as secondary backup which can also be used to restore data to one or more Cassandraclusters. The backup created with semantic de-duplication is fully consistent and deduplicated, meaning only one copy of data is stored in the archival and / or cold storage.

[0050] Semantic de-duplication offers robust de-duplication, although it is resource intensive to track and monitor changes in a Cassandra cluster spanning numerous regions and physical sites and can significantly impact the throughput of a cluster handling write intensive workloads. Further, to achieve Change Data Capture (CDC), node disks need to be dimensioned to handle these extra CDC records, before the clients read and write the data to the archival storage. In the end, the data will still need to be stored in separate Cassandra clusters, which also adds to the total footprint required by the solution.

[0051] Cassandra is a leader-less distributed database and has a storage architecture which makes de-duplication of a backup data a quite difficult task. Here are some factors which adds to the complexity of de-duplication:

[0052] • Each Cassandra SSTable file, can contain its own unique set of data, which might be completely differently structured on a node-by-node basis.

[0053] • Cassandra nodes compact their data independently of each other; hence it just adds to the problem that one key-value pair will be stored in a completely different SSTable on a node-by-node basis.

[0054] • There exists no ability to split one SSTable into key-value pair chunk size, only a fixed size can be chosen. But since key-value pairs can be either bigger or smaller than a pre-defined chunk size, we are not actually able to split the data in a proper way to achieve de-duplication in Cassandra.

[0055] An object of embodiments herein is to improve handling of data in a leaderless distributed database system.

[0056] According to an aspect of embodiments herein, the object is achieved by a method for handling de-duplication in a leaderless distributed database system comprising a set of datacenters, wherein no datacenter in the set of datacenters is configured as a dedicated backup datacenter.

[0057] A datacenter from the set of datacenters in the distributed database system is selected for a keyspace associated with a backup or restore operation.

[0058] A replication factor for the keyspace is obtained.Based on a replication factor for the keyspace, the backup and / or restore operation is assigned to one or more nodes in the selected datacenter.

[0059] The backup and / or restore operation is performed.

[0060] According to an aspect of embodiments herein, the object is achieved by a backup and / or restore controller configured to handle de-duplication in a leaderless distributed database system comprising a set of datacenters. No datacenter in the set of datacenters is configured as a dedicated backup datacenter.

[0061] The backup and / or restore controller is configured to select a datacenter from the set of datacenters in the distributed database system for a keyspace associated with a backup or restore operation.

[0062] The backup and / or restore controller is configured to obtain a replication factor for the keyspace.

[0063] Based on the replication factor for the keyspace, the backup and / or restore controller is configured to assign the backup and / or restore operation to one or more nodes in the selected datacenter.

[0064] perform the backup and / or restore operation.

[0065] BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Examples of embodiments herein are described in more detail with reference to attached drawings in which:

[0067] Figure 1 illustrates an example according to prior art.

[0068] Figure 2 illustrates an example according to prior art.

[0069] Figure 3 illustrates an example according to prior art.

[0070] Figure 4 illustrates an example according to prior art.

[0071] Figure 5 illustrates an example according to prior art.

[0072] Figure 6 illustrates an example according to examples of embodiments herein.

[0073] Figure 7 illustrates an example according to examples of embodiments herein.

[0074] Figure 8 illustrates an example according to examples of embodiments herein.

[0075] Figure 9 is a flowchart depicting embodiments of a method in a backup and / or restore controller.

[0076] Figure 10 illustrates an example according to embodiments herein.

[0077] Figure 11 is a schematic block diagram illustrating embodiments of a backup and / or restore controller.Figure 12 shows an example of a communication system QQ100 in accordance with some embodiments.

[0078] Figure 13 shows a UE QQ200 in accordance with some embodiments.

[0079] Figure 14 shows a network node QQ300 in accordance with some embodiments.

[0080] Figure 15 is a block diagram illustrating a virtualization environment QQ400 in which functions implemented by some embodiments may be virtualized.

[0081] DETAILED DESCRIPTION

[0082] Embodiments herein relate to de-duplication in a leaderless distributed database system.

[0083] Cassandra database system will be used as an example of database system herein. The embodiments described below are, however not limited only to Cassandra, but may use together with any database system.

[0084] The high-level architecture to address the shortcomings of the existing solution is shown in Figure 6.

[0085] Here the following components are depicted:

[0086] • Backup Restore (BR) controller- A Command Line Interface (CLI) to initiate either a backup or restore operation. This layer can be seen as an orchestration layer. The controller will spawn BR operation jobs which will execute on designated database nodes.

[0087] • Backup Restore (BR) operation job - a job responsible for performing tasks required for the backup or restore operations

[0088] • Database - The distributed database which also contains software for downloading / uploading data to and from a cold storage like Amazon S3. Further it contains software to stream backup data to the correct replicas. In addition to this the database has a data disk, depicted as a Physical Volume Claim (PVC) in a Kubernetes environment.

[0089] • S3 - An external backend used to store the backups for the distributed database. Amazon S3 is used here as an example of cold storage. And other cold storages can be used.

[0090] The core essentials of this backup restore solution are the optimization techniques employed to both reduce the cold storage footprint, and at the same time does not require the addition of a dedicated backup restore datacenter to solve the de-duplication problem.In addition, table parallelism is employed to achieve significant performance benefits. This is done by designating keyspace / table responsibility for certain database racks in the system, to handle either the backup or restore operation.

[0091] The optimization technique is achieved through two layers of de-duplication:

[0092] • Using the partition configuration, one datacenter is selected for either a backup or restore. This selection is done on a keyspace level basis.

[0093] • Within the selected datacenter, one rack will be selected, as the target rack for either a backup or restore to be performed.

[0094] oThe number of racks selected depends on the replication factor and the number of racks deployed. If the replication factor is equal to the number of racks, only one rack is selected as the target rack. Otherwise, a simple majority of the racks must be selected to ensure data-consistency for the BR operation.

[0095] This optimization technique will be called the Backup-Restore Rack (BR-RAC). In Figure 7, an illustration shows which racks (marked with red) are eligible for a BR operation employing the BR-RAC optimization technique.

[0096] The following Table 3 shows the division of work between the elected racks in the database system. The table will indicate a mapping between what rack has the responsibility for what keyspace / table, this is how BR parallelism is achieved without the need for multi-threading within one database node. For simplicity, the keyspaces names will be equal to the partition name.

[0097] The first column shows the selection of:

[0098] • Selected target datacenter

[0099] o Selected rack within that datacenter

[0100] The second column shows what keyspace / table that the rack will be responsible for backing up or performing a restore on.

[0101]

[0102] Table 3Examples of embodiments herein provides:

[0103] • De-duplication using the BR-RAC optimization technique:

[0104] oThis is applicable for both backup & restore;

[0105] o Offers fault tolerance on rack level, and still maintains consistency for backup operations.

[0106] o This is achieved without any additional footprint requirements, compared to existing technologies

[0107] • Partition configuration is employed to make sure we take a backup / restore of active datacenters, to ensure consistency of the backup:

[0108] o This allows us to deduplicate on two levels: datacenter and rack level.

[0109] • Provide an optimized way to use an external backend, employing techniques such as preemptive download;

[0110] • Offers various filter functionality:

[0111] okeyspace, table level filters;

[0112] olnclusions and exclusions filters.

[0113] Partition configuration

[0114] In the complex installation of a disturbed database, data sharding or partitioning of your application data is a must. In telecom industry in this case, this partitioning is achieved through a partitioning configuration file, which allows the customer to decide replication for the following data characteristics:

[0115] • The type of data stored:

[0116] o Global data - Data which needs to be accessed on several or all sites, to avoid any cross-site latency. This is achieved by replicating the data to one datacenter per site;

[0117] o Partitioned data - Data which needs to be accessed only in the local site and can be replicated to one additional site for geographical redundancy purposes.

[0118] • If the data should be replicated only on the local site

[0119] • How many partitions / shards to have for the data. One table might be chosen to be sharded multiple times, to avoid having too large tables in the system. This sharding of data can only be done on data types which fall into the “partitioned data” type category.The partition configuration is used as input to achieve de-duplication for both the backup and restore operation. A more detailed representation of the partition configuration is shown in Table 4.

[0120]

[0121] Table 4

[0122] For Table 4, here are the detailed explanations of each column:

[0123] • Keyspace name: The Cassandra keyspace name, which can contain one or many tables within in. The replication factor is configured on the keyspace level (which can also be seen on the last column “Replication” in the table) • Active datacenter: The datacenter on which database applications perform active writes, reads and deletes towards. This from an application standpoint means, the active datacenter has the latest information, and can be fully relied upon when it comes to consistency.

[0124] • Passive datacenter: The datacenter kept up to date using the replication mechanism for all operation done towards the active datacenter. Since the replication mechanism is an async operation, the “passive datacenter” will not have the latest data in real time and will lag the “active datacenter” in terms of data consistency. The primary purpose of the “passive datacenter” is to have a standby datacenter in case the active datacenter has failed. • Replication: the replication factor set for the specified keyspace. This will determine which datacenters will contain the data for the keyspace, and how many replicas will contain the data within each specified datacenter.

[0125] Partition configuration multi-dimensional

[0126] The partition configuration is a multi-dimensional configuration, which can divide data into several layers per client application:

[0127] • First, data can be divided per distinct application component;• Second, the data can then further be divided into datatype being stored by the application, if its customer related data, configuration, historical data, etc;

[0128] • Thirdly, it can be divided into if the data is of GLOBAL or PARTITION type.

[0129] By making use of the partition configuration this allows the invention to backup or restore on any of the above-mentioned data layers, using the filtering mechanism.

[0130] BR-RAC

[0131] The partitioning configuration in conjunction with Cassandra cluster topology is leveraged to distribute BR operation workload among datacenters of the Cassandra cluster deployed across many physical sites.

[0132] Distribution of workload enables parallelism where each datacenter will execute BR operation on a unique set of keyspaces assigned only to itself, and offer fault isolation, meaning in event of a BR operation failure on any given site, the failed site can be safety retried without spreading the affect to other sites.

[0133] The distribution is applied per keyspace, where for each keyspace:

[0134] • The responsible datacenter is identified using the partitioning configuration;

[0135] • Within the selected datacenter, the number of racks in the datacenter is evaluated against the keyspace replication factor to profile the data ownership among the racks;

[0136] • If the number of racks in the selected datacenter is equal to the replication factor of the keyspace in the datacenter, all pods in only one of the racks are chosen to perform BR operation. Otherwise, the simple majority of racks [(rack count / 2) + 1] are selected to ensure data-consistency for BR operation.

[0137] • Additionally, if only a single rack exists with number of replicas in the rack equal to replication factor of the keyspace in the selected datacenter, only one replica in the rack is chosen to perform BR operation given that all replicas will have one copy of the data stored in them.

[0138] Below is the pseudo code for keyspace distribution:

[0139]

[0140]

[0141] Operation specific optimizations

[0142] Given the difference in actions performed between a backup operation and a restore operation, additional optimizations at operation level were implemented to enhance the performance and robustness of the operations.

[0143] Backup-specific optimizations

[0144] Toleration to node / rack unavailability: When racks to participate in backup from a datacenter are selected during keyspace distribution to datacenters, the racks are filtered based on if all nodes in the rack are in UN state in the Cassandra Cluster. If all nodes in the rack are not UN, the next subsequent rack is selected thereby offering fault tolerance in events of rack failure. Such functionality allows the operation engineer to recover the failed rack without fearing losing backup or missing a scheduled backup cycle. It also minimizes the cost of storing data on external storages by storing de-duplicated copy of data.

[0145] Restore-specific optimizations

[0146] Preemptive Downloads: Each table in keyspace is preemptively downloaded factoring in the available disk space in order to reduce wait time for fetching files from the cold storage thereby speeding the restore. When a table is downloaded, it is stored in a temporary directory from where it will stream to Cassandra nodes in one or many datacenters in the cluster who owns the entry in data depending on the token ranges assigned to each node. The table data in the temporary folder is cleaned after streamingto make space for other tables to be restored. As depicted in Figure 8, sequential downloaded are blocked by stream completion, not accounting for the available sufficient disk storage to download the next table thereby increasing total restore time. In the example below, preemptive downloads take 40% less time to complete the restore operation given tables are downloaded in the background when another table is streamed if sufficient disk exists to support both streaming and downloading activities.

[0147] Preemptive Compactions: To ensure disk health during a restore operation, preemptive compactions are run on keyspaces that are been restored in the background.

[0148] Table Slicing: Large tables are downloaded and streamed in parts of smaller sizes to control the maximum amount of data streamed in the cluster at any given instance and mitigate the risk of filling the disk during the restore.

[0149] The above-described problem is addressed in a number of embodiments, some of which may be seen as alternatives, while some may be used in combination.

[0150] A method according to embodiments will now be described together with Figure 9.

[0151] Figure 9 shows example embodiments of a method for handling de-duplication in the leaderless distributed database system 100 comprising a set of datacenters. No datacenter 110 in the set of datacenters is configured as a dedicated backup datacenter.

[0152] The method may e.g., be performed by a backup and / or restore controller 101 comprised in, connected to, the database system 100. The method comprises the following actions, which actions may be taken in any suitable order. Actions that are optional are presented in dashed boxes in Figure 9.

[0153] Action 901

[0154] A datacenter 110 is selected from the set of datacenters in the distributed database system 100. The datacenter 110 is selected for a keyspace associated with a backup or restore operation. In other words, the datacenter 110 is selected for the backup and / or restore operation. A backup operation may e.g., comprise performing a backup of the keyspace. A restore operation may e.g., comprise restoring a backed up version of the keyspace. The selected datacenter 110 may e.g., comprise the datacenter where the keyspace is to be backed up to, and / or the datacenter the keyspace is to be restored to.

[0155] In some embodiments, each datacenter comprises a set of nodes. The set of nodes may be group into one or more racks. This may e.g., mean that a datacenter, such as the selected datacenter 110, comprises one or more racks, where each rack comprises a number of nodes.In some embodiments, each rack comprises a replica of data associated with the keyspace. The data may be distributed between the nodes in the rack.

[0156] In some examples, all nodes in a rack comprises a replica of the keyspace data. This may e.g., be when the datacenter comprises one rack, and the number of nodes in the rack equals the replication factor. For a backup operation, any one of the nodes in the rack may selected to be used as a backup. For a restore operation, the data will be restored to all nodes in the rack. This may e.g., mean that, during the restore operation, the keyspace is streamed to one node in the rack. The keyspace will then be stream to the remaining nodes in the rack, i.e. , to the replicas comprised in the remaining nodes.

[0157] In some examples, all racks in a data center comprises a replica of the keyspace data, and data is distributed among the nodes in the rack. This may e.g., be when the number of racks in the datacenter equals the replication factor. For a backup operation, one of the racks may be selected to use as a backup. For a restore operation, the data will be restored to all racks in the datacenter. This may e.g., mean that, during the restore operation, the keyspace is streamed to one of the racks. The keyspace will then be stream to the remaining racks, i.e., to the replicas comprised in the remaining racks.

[0158] In some embodiments, a rack comprises a replica of data associated with the keyspace, and wherein the data is distributed between the nodes in the rack. This may e.g., mean that the datacenter comprises two or more racks and that only at least one rack, but not all racks, comprises a replica of the data.

[0159] In some embodiments, the keyspace is associated with a configuration. The configuration indicates any one or more out of one or more datacenters the keyspace is replicated to, and a datacenter associated with the keyspace to be backed up. This may e.g., mean that the configuration indicates a datacenter where a replica of keyspace is comprised. Alternatively, or additionally, this may e.g., mean that the configuration indicates a datacenter eligible for backup, such as which datacenter to back up data from.

[0160] In some embodiments, the datacenter is selected from the configuration associated with the keyspace. In other words, the datacenter is selected based on configuration, e.g., by selecting a datacenter indicated in the configuration.

[0161] In some embodiments, responsive to detecting that a datacenter indicated in the configuration is unavailable, a datacenter comprising a replica of the keyspace is selected. The data center comprising the replica may e.g., be indicated in the configuration.

[0162] Action 902A replication factor for the keyspace is obtained. The replication factor may e.g., be obtained from the configuration associated with the keyspace.

[0163] Action 903

[0164] Based on a replication factor for the keyspace, the backup and / or restore operation is assigned to one or more nodes in the selected datacenter 110.

[0165] In some embodiments, assigning the backup and / or restore operation comprises assigning the backup and / or restore operation to one or more nodes in one or more racks. In one example, the backup and / or restore operation may be assigned to one node in one rack. In another example, the backup and / or restore operation may be assigned to all nodes in one rack. In yet another examples, the backup and / or restore operation may be assigned to all nodes in one or a majority of the racks.

[0166] Action 904

[0167] The backup and / or restore operation is performed.

[0168] In some embodiments, the restore operation comprises performing a preemptive restore operation. The preemptive restore operation may e.g., mean that the restore operation is performed in a way that reduces the time consumed by the restore operation.

[0169] In some embodiments, the preemptive restore operation comprises iteratively, until all tables in the keyspace are restored:

[0170] R1) Download a first table, or a first part of the first table, associated with the keyspace.

[0171] R2) Stream the first table, or the first part of the first table, to the selected datacenter 110 according to the assignment.

[0172] R3) While streaming the first table, or the first part of the first table, download a subsequent table, a subsequent part of the first table or a first part of the subsequent table, associated with the keyspace.

[0173] R4) Responsive to the streaming of the first table, or first part of the first table, being completed, stream the subsequent table, subsequent part of the first table or first part of the subsequent table, to the selected datacenter 110 according to the assignment.

[0174] In other words, once the first table, or part of the first table, is downloaded and streaming of said first table, or part of first table, has commenced, a download of the second table, or second part of the first table, or even a first part of the second table. This way the time consumed for performing the backup operation will be reduced compared to when the restore operation is performed sequentially.In an example, where the keyspace comprises 3 tables, the preemptive restore operation may be described as follows:

[0175] i) Download table 1.

[0176] ii) Stream table 1.

[0177] iii) While streaming table 1, download table 2,

[0178] iv) When streaming of table 1 is complete, stream table 2.

[0179] v) While streaming table 2, download table 3.

[0180] vi) When streaming of table 2 is complete, stream table 3.

[0181] In some examples, there may be an interruption time between the completion of streaming a table and starting to stream the next table if the download of the next table has net been completed. In some examples, this may be overcome by starting streaming of the next table before the download is completed.

[0182] The tables may e.g., be downloaded to, and stream from the backup and / or restore controller 101.

[0183] In some embodiments, prior to downloading a table, a size of the table is evaluated. Upon determining that the size of the table is larger than a threshold, the table may be sliced into at least two parts. The parts are smaller than, or equal to, the threshold. This may allow the preemptive restore operation to be performed more efficiently. This since the storage space of the backup and / or restore controller 101 may be limited, and threshold may be set of allow the downloading of a second part of a table while a first part of the table is streamed, and thus already stored in the storage of the backup and / or restore controller 101.

[0184] In some embodiments, the backup operation comprises:

[0185] B1) Determine, for each rack in the selected datacenter 110, whether all nodes in the rack are indicated as Up and Normal (UN).

[0186] B2) Responsive to at least one node in a first rack being indicated as not being UN, select a second rack comprising only nodes indicated as UN.

[0187] B3) Responsive to all nodes in all racks being indicated as UN, select the first rack for the backup operation.

[0188] In other words, a rack where all nodes are indicated as UN is always selected for the backup operation.

[0189] In some embodiments, the backup operation comprises applying a backup filter. The filter excludes temporary data from being backed up. This avoids backup of data that does need to be backed up, thus reducing the amount storage space used. The filter may further be used for filtering which tables to backup.Embodiments herein such as the embodiments mentioned above will now be further described and exemplified. The text below is applicable to embodiments herein and may be combined with any suitable embodiment described above.

[0190] Depending on the configuration provided for a BR operation, BR Controller prepares an operation plan which is executed in phases where each phase is a set of one or more Kubernetes jobs executing pre-defined sequence of actions on the Cassandra nodes. The operation plan embeds information about the requested operation kind: backup or restore, Cassandra topology, impacted keyspaces and / or tables in data, and jobs manifests that will perform each phase of operation. BR Controller orchestrates jobs in each phase and is the point of interaction for the end user to trigger BR operations. BR Controller implements the concept of BR-RAC, to achieve data de-duplication and coordinates the work distribution among jobs using Kubernetes config maps.

[0191] BR Controller supports BR operations using S3 protocol storage backends including Amazon Web Services (AWS) S3, and Minio. With that said, this approach of backup and restores can be extended to work in conjunction with other forms of external and public object storages as well as a shared storage provisioned on-premises through affordable bulk stores like hard disks. When a BR operation is performed, the data is stored to or is retrieved from the configured S3 protocol storage backend provided using the location information config map in the same namespace as where the BR Controller is deployed in the Kubernetes cluster.

[0192] BR Controller also requires two other config maps called backup data model and restore data model. As the name suggests, backup data model defines the keyspaces and / or tables to be included / excluded when performing a backup operation. Similarly, restore configuration limits a restore to keyspaces and / or tables described in the configuration. If the restore configuration is empty, all keyspaces and tables as in the backup will be restored. Both backup and restore configurations support regex, meaning keyspaces and / or tables can be included / excluded based on string patterns as well.

[0193] BR Controller also establishes Cassandra Query Language (CQL) connection with the Cassandra cluster on which the BR operations will be performed to derive the Cassandra topology and partitioning configuration, to prepare operation plan using the BR-RAC concept, as discussed above.

[0194] A BR job is a subset of Kubernetes jobs spawned to complete a phase in a BR operation. Each job is designated one Cassandra node to perform the BR operation on.Number of jobs required to complete the operation is derived using the BR-RAC concept. In most cases, the required number of jobs is less than the total number of Cassandra nodes in the cluster for the BR operation. Meaning, BR operation will be performed only on selected Cassandra nodes while also ensuring that the operation affects all Cassandra nodes in the Cassandra cluster. BR jobs interact with the configured S3 protocol storage backend to store or retrieve the data.

[0195] There are four phases of each BR operation namely import-schemas, prepare, execute and cleanup, respectively.

[0196] Import schemas prepare the operation plan for the BR job using the partition and replication configuration retrieved from the distributed database system.

[0197] Prepare phase runs validations and housekeeping on the backups stored in configured S3 protocol storage backend prior to commencing the execution of BR operation and alters configuration of participating nodes to optimally support the BR operation.

[0198] After the prepare phase, the execute phase is performed where the data is backed from or restored to participating nodes.

[0199] Lastly, in cleanup phase, the operation success is validated by examining the data and all altered Cassandra configurations are reset to their original state.

[0200] Cassandra node restart is not required in any of the phases of a BR operation allowing vendors to continue performing read and write operations on non-impacted keyspaces and / or tables by the BR operation.

[0201] The BR controller's deployment in a deployment based on virtual machines will look very similar to the containerized environment. Main difference being that the concept of “jobs” does not exist, as this is a Kubernetes specific feature. This can however be replaced with the introduction of ansible role framework, which can be used to achieve a similar mechanism.

[0202] In addition to this, in a virtual deployment each component depicted in Figure 10 will have its own dedicated:

[0203] • CPU

[0204] • Memory

[0205] • Disk

[0206] • Operating system (OS)

[0207] Figure 14 depicts an End-to-End Flow of Backup and Restore in a Deployment Based in Virtual machines.In Figure 10, an end-to-end flow is depicted for the backup restore solution in a virtual environment:

[0208] 1. User triggers either a backup or restore operation:

[0209] a. From the CLI chose to trigger either a backup or restore (default preconfigured values are available for optimization). User can also select to do a partial restore / backup with exclusions.

[0210] b. BR-controller optimizes the datacenter and rack selection based on the partition configuration and executes backup / restore commands to the selected database nodes.

[0211] 2. The backup / restore command reaches the selected database nodes: a. Perform internal backup or restore logic.

[0212] b. If a restore is preformed, preemptive download logic is triggered.

[0213] 3. Download files from S3 (if restore), or upload files to S3 (if backup).

[0214] 4. Respond back if the backup / restore was successful to end user.

[0215] According to examples of embodiments herein, this disclosure aims to enhance the existing solutions which are provided by any vendor for backup restore solutions.

[0216] Examples of embodiments herein aim at keeping total cost ownership (TCO) low with only an addition of a backup and / or controller service.

[0217] Examples of embodiments herein aim at lowering the costs of storing backups through de-duplication, allowing up to 80% cold storage savings.

[0218] A backup and / or restore operation may e.g., be invoked only on designated nodes, thereby adding less pressure to the system.

[0219] Examples of embodiments herein aim for optimal recovery time objective (RTO). This may e.g., be achieved by:

[0220] Backup and / restore optimization techniques to only restore unique data and never duplicate in the best-case scenarios. This may e.g., be achieved by: Filter out any temporary data.

[0221] Use preemptive download to avoid long wait time for external storages.

[0222] These two objectives in combination are not achieved by any solution available today within a leaderless distributed database.

[0223] According to examples of embodiments, methods of de-duplicating backup / restore data in a leaderless distributed database system comprising of a backup and / or restore controller associated with the database system is provided.A leaderless database may comprise N number of nodes

[0224] No node will be the leader / owner of the data, hence the data will be scattered equally among the nodes in the cluster. Any node in the cluster can handle a write / read request.

[0225] No node within the database has the same data. Below is an example showing 6 nodes, and illustrates which node has the data “X”, “Y” and “Z”

[0226]

[0227] No data within the nodes will be structured the same way as data is independently compacted on a node-by-node basis.

[0228] According to examples of embodiments herein a keyspace and / or partitioning configuration is extended.

[0229] The partition configuration will let us know on which datacenters, comprising N nodes, keyspaces are replicated on. All datacenters in the keyspace and / or partition configuration can be active

[0230] The partition configuration can divide data according to application datatypes; configuration data, historical data, search data etc.

[0231] The partition configuration is extended with additional fields to indicate which datacenters are eligible for backup.

[0232] In some examples, s new client-side routing layer is introduced to indicate what datacenter should be backed up. This routing configuration is used by the backup and / or restore controller. This is because there might be multiple active datacenters to store the data, to restrict this the operator can configure, what will be the primary datacenter for backups. If backup datacenters are not configured, the backup and / or restore controller will by default select first “active” datacenter based on latency to be the backup datacenter.

[0233]

[0234] Some examples of embodiments herein make use of the rack concept within the distributed leaderless database.

[0235] The rack concept will uniquely divide data between racks created. Hence no rack will have redundant data within the racks.

[0236]

[0237] Still no node has all the data, the data is still scattered but evenly divided between the racks

[0238] According to examples of embodiments herein, de-duplication of the data may comprise selecting one of the datacenters based on few factors. The same data might be replicated on a numbers of datacenters. The factors may e.g., comprise:

[0239] Distance to datacenter. By default the optimization logic selects a datacenter on a site basis. Each backup and / or restore controller, which is deployed on a sitebasis, may be responsible for selecting a datacenter to perform a backup / restore on the site its deployed on.

[0240] If its an active datacenter, writes will only be done on the active datacenters. After one active / backup datacenter has been selected, the de-duplication logic further deep divides in the rack deployment, based on e.g.,:

[0241] If the number of racks deployed matches the replication factor set for the datacenter, then a full de-duplication is possible.

[0242] If the number of racks is not equal to the replication factor, then a full deduplication is not possible. A backup / restore of a majority of the racks within the datacenter may be performed. Still saving 80% disk using this approach, assuming there is 3 nodes per datacenter which contains the data, and data is scattered to 3 datacenters.

[0243] According to examples of embodiments herein, backup and / or restore optimization is able to use an external backend in a seamless fashion, e.g., by:

[0244] Employing a preemptive download before starting the next restore sequence. Before an preemptive download of a table starts, it will check that disk storage is sufficient for temporarily storing the table and has space to stream the table to all nodes who need it.According to examples of embodiments herein, backup and / or restore optimizations to filter out temporary data / tables from being backed up may be achieved.

[0245] This is done by exposing an exclusion filter, which can be set for each application in the system. This leads for a telecom system up to a reduction of 88% of data to be backed up.

[0246] According to examples of embodiments herein, backup and / or restore optimization is also able to handle topology changes through its streaming usage for restore operations. De-duplication is still possible during topology changes. Topology changes may e.g., comprise:

[0247] Nodes being added or removed.

[0248] Datacenters changing name.

[0249] According to examples of embodiments herein, backup and / or restore optimization may also be able to handle multiple database clusters, where application data is divided to isolated systems. One such use case is the Mobile virtual network operator (MVNO).

[0250] This is done by further enhancing the partition configuration to indicate on what clusters, such as datacenter, data should be stored.

[0251] According to examples of embodiments herein, backup and / or restore optimization can automatically handle failover and still perform a backup if e.g., a datacenter goes down and / or a rack within a datacenter goes down. This requires no manual intervention as the BR controller will automatically select the next datacenter in line, which holds a passive replica.

[0252] To perform the method actions above, the backup and / or restore controller 101 is configured to handle de-duplication in a leaderless distributed database system 100 comprising a set of datacenters. The backup and / or restore controller 101 may comprise an arrangement depicted in Figure 11.

[0253] The backup and / or restore controller 101 may comprise an input and output interface 30 configured to communicate with each other. The input and output interface 30 may comprise a receiver, e.g. wired and / or wireless, (not shown) and a transmitter, e.g. wired and / or wireless, (not shown).

[0254] The embodiments herein may be implemented through a respective processor or one or more processors, such as at least one processor 31 of a processing circuitry in the backup and / or restore controller 101 depicted in Figure 15, together with computer program code for performing the functions and actions of the embodiments herein. The program code mentioned above may also be provided as a computer program product, forinstance in the form of a data carrier carrying computer program code for performing the embodiments herein when being loaded into the backup and / or restore controller 101. One such carrier may be in the form of a CD ROM disc. It is however feasible with other data carriers such as a memory stick. The computer program code may furthermore be provided as pure program code on a server and downloaded to the backup and / or restore controller 101.

[0255] The backup and / or restore controller 101 and / or processor 31 is configured to handle de-duplication in a leaderless distributed database system 100 comprising a set of datacenters. No datacenter 110 in the set of datacenters is configured as a dedicated backup datacenter.

[0256] The backup and / or restore controller 101 and / or processor 31 is configured to select a datacenter 110 from the set of datacenters in the distributed database system 100 for a keyspace associated with a backup or restore operation.

[0257] The backup and / or restore controller 101 and / or processor 31 is configured to obtain a replication factor for the keyspace.

[0258] Based on a replication factor for the keyspace, the backup and / or restore controller 101 and / or processor 31 is configured to assign the backup and / or restore operation to one or more nodes in the selected datacenter 110.

[0259] The backup and / or restore controller 101 and / or processor 31 is configured to perform the backup and / or restore operation.

[0260] In some embodiments, each datacenter comprises a set of nodes, and where the set of nodes are group into one or more racks.

[0261] In some embodiments, each rack comprises a replica of data associated with the keyspace, and wherein the data is distributed between the nodes in the rack.

[0262] In some embodiments, a rack comprises a replica of data associated with the keyspace, and wherein the data is distributed between the nodes in the rack.

[0263] In some embodiments, the backup and / or restore controller 101 and / or processor 31 is configured to assign the backup and / or restore operation by further being configured to assign the backup and / or restore operation to one or more nodes in one or more racks.

[0264] In some embodiments, the restore operation comprises performing a preemptive restore operation.

[0265] In some embodiments, the preemptive restore operation comprises iteratively, until all tables in the keyspace are restored:

[0266] download a first table, or a first part of the first table, associated with the keyspace,stream the first table, or the first part of the first table, to the selected datacenter 110 according to the assignment,

[0267] while streaming the first table, or the first part of the first table, download a second table, or a second part of the first table, associated with the keyspace, and responsive to the streaming of the first table, or first part of the first table, being completed, stream the second table, or second part of the first table, to the selected datacenter 110 according to the assignment.

[0268] In some embodiments, prior to downloading a table, evaluate a size of the table. Upon determining that the size of the table is larger than a threshold, slice the table into at least two parts. The parts are smaller than, or equal to, the threshold.

[0269] In some embodiments, the backup operation comprises:

[0270] determine, for each rack in the selected datacenter 110, whether all nodes in the rack are indicated as Up and Normal, UN, and

[0271] responsive to at least one node in a first rack being indicated as not being UN, selecting a second rack comprising only nodes indicated as UN.

[0272] In some embodiments, the backup operation comprises applying a backup filter, which filter excludes temporary data from being backed up.

[0273] In some embodiments, the keyspace is associated with a configuration, which configuration indicates any one or more out of:

[0274] - one or more datacenters the keyspace is replicated to, and

[0275] - a datacenter associated with the keyspace to be backed up.

[0276] In some embodiments, the datacenter is selected from the configuration associated with the keyspace.

[0277] The backup and / or restore controller 101 may further comprise respective a memory 32 comprising one or more memory units. The memory 32 comprises instructions executable by the processor 31 in the backup and / or restore controller 101.

[0278] The memory 32 is arranged to be used to store instructions, data, configurations, handovertiming conditions, handovertiming condition data, requests, responses, messages, identifiers, indications, parameters, applications to perform the methods herein when being executed in the backup and / or restore controller 101.

[0279] In some embodiments, a computer program 33 comprises instructions, which when executed by the at least one processor 31, cause the at least one processor 31 of the backup and / or restore controller 101 to perform the actions above.

[0280] In some embodiments, a respective carrier 34 comprises the respective computer program 33, wherein the carrier 34 is one of an electronic signal, an optical signal, anelectromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal, or a computer-readable storage medium.

[0281] Thus, embodiments herein may disclose the backup and / or restore controller 101 e.g., configured to handle de-duplication in the leaderless distributed database system 100 comprising a set of datacenters. No datacenter 110 in the set of datacenters is configured as a dedicated backup datacenter. The backup and / or restore controller 101 comprises the processor 31 and the memory 32, said memory 32 comprising instructions executable by said processor 31 whereby said backup and / or restore controller 101 is operative to perform any of the methods herein.

[0282] As will be readily understood by those familiar with communications design, that functions means or modules may be implemented using digital logic and / or one or more microcontrollers, microprocessors, or other digital hardware. In some embodiments, several or all of the various functions may be implemented together, such as in a single application-specific integrated circuit (ASIC), or in two or more separate devices with appropriate hardware and / or software interfaces between them. Several of the functions may be implemented on a processor shared with other functional components of a base station, for example.

[0283] Alternatively, several of the functional elements of the processing means discussed may be provided through the use of dedicated hardware, while others are provided with hardware for executing software, in association with the appropriate software or firmware. Thus, the term “processor” or “controller” as used herein does not exclusively refer to hardware capable of executing software and may implicitly include, without limitation, digital signal processor (DSP) hardware, read-only memory (ROM) for storing software, random-access memory for storing software and / or program or application data, and nonvolatile memory. Other hardware, conventional and / or custom, may also be included. Designers of communications receivers will appreciate the cost, performance, and maintenance trade-offs inherent in these design choices.

[0284] Any appropriate steps, methods, features, functions, or benefits disclosed herein may be performed through one or more functional units or modules of one or more virtual apparatuses. Each virtual apparatus may comprise a number of these functional units. These functional units may be implemented via processing circuitry, which may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include digital signal processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM),random-access memory (RAM), cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory includes program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein. In some implementations, the processing circuitry may be used to cause the respective functional unit to perform corresponding functions according one or more embodiments of the present disclosure.

[0285] ADDITIONAL EXPLANATION

[0286] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0287] Figure 12 shows an example of a communication system QQ100 in accordance with some embodiments.

[0288] In the example, the communication system QQ100 includes a telecommunication network QQ102 that includes an access network QQ104, such as a radio access network (RAN), and a core network QQ106, which includes one or more core network nodes QQ108. The access network QQ104 includes one or more access network nodes, such as network nodes QQ110a and QQ110b (one or more of which may be generally referred to as network nodes QQ110), or any other similar 3rdGeneration Partnership Project (3GPP) access nodes or non-3GPP access points. Moreover, as will be appreciated by those of skill in the art, a network node is not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it will be understood that network nodes include disaggregated implementations or portions thereof. For example, in some embodiments, the telecommunication network QQ102 includes one or more Open-RAN (ORAN) network nodes. An ORAN network node is a node in the telecommunication network QQ102 that supports an ORAN specification (e.g., a specification published by the O-RAN Alliance, or any similar organization) and may operate alone or together with other nodes to implement one or more functionalities of any node in the telecommunication network QQ102, including one or more network nodes QQ110 and / or core network nodes QQ108.

[0289] Examples of an ORAN network node include an open radio unit (O-RU), an open distributed unit (O-DU), an open central unit (O-CU), including an O-CU control plane (O-CU-CP) or an O-CU user plane (O-CU-UP), a RAN intelligent controller (near-real time or non-real time) hosting software or software plug-ins, such as a near-real time controlapplication (e.g., xApp) or a non-real time control application (e.g., rApp), or any combination thereof (the adjective “open” designating support of an ORAN specification). The network node may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an A1, F1, W1, E1, E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Moreover, an ORAN access node may be a logical node in a physical node. Furthermore, an ORAN network node may be implemented in a virtualization environment (described further below) in which one or more network functions are virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a Service Management and Orchestration Framework via an 0-2 interface defined by the O-RAN Alliance or comparable technologies. The network nodes QQ110 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs QQ112a, QQ112b, QQ112c, and QQ112d (one or more of which may be generally referred to as UEs QQ112) to the core network QQ106 over one or more wireless connections.

[0290] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system QQ100 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system QQ100 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0291] The UEs QQ112 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes QQ110 and other communication devices. Similarly, the network nodes QQ110 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs QQ112 and / or with other network nodes or equipment in the telecommunication network QQ102 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network QQ102.

[0292] In the depicted example, the core network QQ106 connects the network nodes QQ110 to one or more host computing systems, such as host QQ116. These connections may be direct or indirect via one or more intermediary networks or devices. In otherexamples, network nodes may be directly coupled to hosts. The core network QQ106 includes one more core network nodes (e.g., core network node QQ108) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node QQ108. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (ALISF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0293] The host QQ116 may be under the ownership or control of a service provider other than an operator or provider of the access network QQ104 and / or the telecommunication network QQ102. The host QQ116 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0294] As a whole, the communication system QQ100 of Figure 12 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0295] In some examples, the telecommunication network QQ102 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network QQ102 may support network slicing to provide different logical networks to differentdevices that are connected to the telecommunication network QQ102. For example, the telecommunications network QQ102 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC) / Massive loT services to yet further UEs.

[0296] In some examples, the UEs QQ112 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network QQ104 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network QQ104. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[0297] In the example, the hub QQ114 communicates with the access network QQ104 to facilitate indirect communication between one or more UEs (e.g., UE QQ112c and / or QQ112d) and network nodes (e.g., network node QQ110b). In some examples, the hub QQ114 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub QQ114 may be a broadband router enabling access to the core network QQ106 for the UEs. As another example, the hub QQ114 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes QQ110, or by executable code, script, process, or other instructions in the hub QQ114. As another example, the hub QQ114 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub QQ114 may be a content source. For example, for a UE that is a VR device, display, loudspeaker, or other media delivery device, the hub QQ114 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub QQ114 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub QQ114 acts as a proxy server or orchestrator for the UEs, in particular if one or more of the UEs are low energy loT devices.

[0298] The hub QQ114 may have a constant / persistent or intermittent connection to the network node QQ110b. The hub QQ114 may also allow for a different communicationscheme and / or schedule between the hub QQ114 and UEs (e.g., UE QQ112c and / or QQ112d), and between the hub QQ114 and the core network QQ106. In other examples, the hub QQ114 is connected to the core network QQ106 and / or one or more UEs via a wired connection. Moreover, the hub QQ114 may be configured to connect to an M2M service provider over the access network QQ104 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes QQ110 while still connected via the hub QQ114 via a wired or wireless connection. In some embodiments, the hub QQ114 may be a dedicated hub - that is, a hub whose primary function is to route communications to / from the UEs from / to the network node QQ110b. In other embodiments, the hub QQ114 may be a non-dedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node QQ110b, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0299] Figure 13 shows a UE QQ200 in accordance with some embodiments. The UE QQ200 presents additional details of some embodiments of the UE QQ112 of Figure 1. As used herein, a UE refers to a device capable, configured, arranged and / or operable to communicate wirelessly with network nodes and / or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage / playback device, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), an Augmented Reality (AR) or Virtual Reality (VR) device, wireless customer-premise equipment (CPE), vehicle, vehiclemounted or vehicle embedded / integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB-loT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0300] A UE may support device-to-device (D2D) communication, for example by implementing a 3GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle-to-everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a UE may represent a device that is notintended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[0301] The UE QQ200 includes processing circuitry QQ202 that is operatively coupled via a bus QQ204 to an input / output interface QQ206, a power source QQ208, a memory QQ210, a communication interface QQ212, and / or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in Figure 13. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0302] The processing circuitry QQ202 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory QQ210. The processing circuitry QQ202 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry QQ202 may include multiple central processing units (CPUs).

[0303] In the example, the input / output interface QQ206 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE QQ200. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.In some embodiments, the power source QQ208 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source QQ208 may further include power circuitry for delivering power from the power source QQ208 itself, and / or an external power source, to the various parts of the UE QQ200 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source QQ208. Power circuitry may perform any formatting, converting, or other modification to the power from the power source QQ208 to make the power suitable for the respective components of the UE QQ200 to which power is supplied.

[0304] The memory QQ210 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory QQ210 includes one or more application programs QQ214, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data QQ216. The memory QQ210 may store, for use by the UE QQ200, any of a variety of various operating systems or combinations of operating systems.

[0305] The memory QQ210 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUlCC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory QQ210 may allow the UE QQ200 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory QQ210, which may be or comprise a device-readable storage medium.The processing circuitry QQ202 may be configured to communicate with an access network or other network using the communication interface QQ212. The communication interface QQ212 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna QQ222. The communication interface QQ212 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a network node in an access network). Each transceiver may include a transmitter QQ218 and / or a receiver QQ220 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter QQ218 and receiver QQ220 may be coupled to one or more antennas (e.g., antenna QQ222) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0306] In the illustrated embodiment, communication functions of the communication interface QQ212 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof.

[0307] Communications may be implemented in according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / internet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0308] Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface QQ212, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to a network node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[0309] As another example, a UE comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator,the motor, or the switch may change. For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0310] A UE, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smartwatch, a fitness tracker, a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an loT device comprises circuitry and / or software in dependence of the intended application of the loT device in addition to other components as described in relation to the UE QQ200 shown in Figure 13.

[0311] As yet another specific example, in an loT scenario, a UE may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another UE and / or a network node. The UE may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3GPP NB-loT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.

[0312] In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second UE can also include more than one of the functionalities described above. For example, a UE might comprisethe sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

[0313] Figure 14 shows a network node QQ300 in accordance with some embodiments. As used herein, network node refers to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with a UE and / or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)), O-RAN nodes or components of an O-RAN node (e.g., 0-Rll, 0-Dll, O-CU).

[0314] Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units, distributed units (e.g., in an O-RAN access node) and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[0315] Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).

[0316] The network node QQ300 includes a processing circuitry QQ302, a memory QQ304, a communication interface QQ306, and a power source QQ308. The network node QQ300 may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node QQ300 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In sucha scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node QQ300 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory QQ304 for different RATs) and some components may be reused (e.g., a same antenna QQ310 may be shared by different RATs). The network node QQ300 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node QQ300, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node QQ300.

[0317] The processing circuitry QQ302 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other network node QQ300 components, such as the memory QQ304, to provide network node QQ300 functionality.

[0318] In some embodiments, the processing circuitry QQ302 includes a system on a chip (SOC). In some embodiments, the processing circuitry QQ302 includes one or more of radio frequency (RF) transceiver circuitry QQ312 and baseband processing circuitry QQ314. In some embodiments, the radio frequency (RF) transceiver circuitry QQ312 and the baseband processing circuitry QQ314 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry QQ312 and baseband processing circuitry QQ314 may be on the same chip or set of chips, boards, or units.

[0319] The memory QQ304 may comprise any form of volatile or non-volatile computer-readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry QQ302. The memory QQ304 may store any suitable instructions, data, or information, including a computer program,software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry QQ302 and utilized by the network node QQ300. The memory QQ304 may be used to store any calculations made by the processing circuitry QQ302 and / or any data received via the communication interface QQ306. In some embodiments, the processing circuitry QQ302 and memory QQ304 is integrated.

[0320] The communication interface QQ306 is used in wired or wireless communication of signaling and / or data between a network node, access network, and / or UE. As illustrated, the communication interface QQ306 comprises port(s) / terminal(s) QQ316 to send and receive data, for example to and from a network over a wired connection. The communication interface QQ306 also includes radio front-end circuitry QQ318 that may be coupled to, or in certain embodiments a part of, the antenna QQ310. Radio front-end circuitry QQ318 comprises filters QQ320 and amplifiers QQ322. The radio front-end circuitry QQ318 may be connected to an antenna QQ310 and processing circuitry QQ302. The radio front-end circuitry may be configured to condition signals communicated between antenna QQ310 and processing circuitry QQ302. The radio front-end circuitry QQ318 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry QQ318 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters QQ320 and / or amplifiers QQ322. The radio signal may then be transmitted via the antenna QQ310. Similarly, when receiving data, the antenna QQ310 may collect radio signals which are then converted into digital data by the radio front-end circuitry QQ318. The digital data may be passed to the processing circuitry QQ302. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0321] In certain alternative embodiments, the network node QQ300 does not include separate radio front-end circuitry QQ318, instead, the processing circuitry QQ302 includes radio front-end circuitry and is connected to the antenna QQ310. Similarly, in some embodiments, all or some of the RF transceiver circuitry QQ312 is part of the communication interface QQ306. In still other embodiments, the communication interface QQ306 includes one or more ports or terminals QQ316, the radio front-end circuitry QQ318, and the RF transceiver circuitry QQ312, as part of a radio unit (not shown), and the communication interface QQ306 communicates with the baseband processing circuitry QQ314, which is part of a digital unit (not shown).The antenna QQ310 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. The antenna QQ310 may be coupled to the radio front-end circuitry QQ318 and may be any type of antenna capable of transmitting and receiving data and / or signals wirelessly. In certain embodiments, the antenna QQ310 is separate from the network node QQ300 and connectable to the network node QQ300 through an interface or port.

[0322] The antenna QQ310, communication interface QQ306, and / or the processing circuitry QQ302 may be configured to perform any receiving operations and / or certain obtaining operations described herein as being performed by the network node. Any information, data and / or signals may be received from a UE, another network node and / or any other network equipment. Similarly, the antenna QQ310, the communication interface QQ306, and / or the processing circuitry QQ302 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and / or signals may be transmitted to a UE, another network node and / or any other network equipment.

[0323] The power source QQ308 provides power to the various components of network node QQ300 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source QQ308 may further comprise, or be coupled to, power management circuitry to supply the components of the network node QQ300 with power for performing the functionality described herein. For example, the network node QQ300 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source QQ308. As a further example, the power source QQ308 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0324] Embodiments of the network node QQ300 may include additional components beyond those shown in 15 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the network node QQ300 may include user interface equipment to allow input of information into the network node QQ300 and to allow output of information from the network node QQ300. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node QQ300. In some embodiments providing acore network node, such as core network node 108 of FIG. QQ1, some components, such as the radio front-end circuitry QQ318 and the RF transceiver circuitry QQ312 may be omitted.

[0325] Figure 15 is a block diagram illustrating a virtualization environment QQ400 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments QQ400 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment QQ400 includes components defined by the O-RAN Alliance, such as an O-Cloud environment orchestrated by a Service Management and Orchestration Framework via an 0-2 interface. Virtualization may facilitate distributed implementations of a network node, UE, core network node, or host.

[0326] Applications QQ402 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment Q400 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0327] Hardware QQ404 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers QQ406 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs QQ408a and QQ408b (one or more of which may be generally referred to as VMs QQ408), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer QQ406 may present a virtual operating platform that appears like networking hardware to the VMs QQ408.The VMs QQ408 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer QQ406. Different embodiments of the instance of a virtual appliance QQ402 may be implemented on one or more of VMs QQ408, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0328] In the context of NFV, a VM QQ408 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs QQ408, and that part of hardware QQ404 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs QQ408 on top of the hardware QQ404 and corresponds to the application QQ402.

[0329] Hardware QQ404 may be implemented in a standalone network node with generic or specific components. Hardware QQ404 may implement some functions via virtualization. Alternatively, hardware QQ404 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration QQ410, which, among others, oversees lifecycle management of applications QQ402. In some embodiments, hardware QQ404 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system QQ412 which may alternatively be used for communication between hardware nodes and radio units.

[0330] Although the computing devices described herein (e.g., UEs, network nodes) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methodsdisclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0331] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer-readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0332] When using the word "comprise" or “comprising” it shall be interpreted as nonlimiting, i.e. meaning "consist at least of".

[0333] The embodiments herein are not limited to the preferred embodiments described above. Various alternatives, modifications and equivalents may be used.

Claims

CLAIMS1. A method for handling de-duplication in a leaderless distributed database system (100) comprising a set of datacenters, wherein no datacenter (110) in the set of datacenters configured as a dedicated backup datacenter, the method comprising:selecting (901) a datacenter (110) from the set of datacenters in the distributed database system (100) for a keyspace associated with a backup or restore operation, obtaining (902) a replication factor for the keyspace,based on a replication factor for the keyspace, assigning (903) the backup and / or restore operation to one or more nodes in the selected datacenter (110)performing (904) the backup and / or restore operation.

2. The method according to claim 1 , wherein each datacenter comprises a set of nodes, and where the set of nodes are grouped into one or more racks.

3. The method according to claim 2, wherein each rack comprises a replica of data associated with the keyspace, and wherein the data is distributed between the nodes in the rack.

4. The method according to claim 2, wherein a rack comprises a replica of data associated with the keyspace, and wherein the data is distributed between the nodes in the rack.

5. The method according to any of claims 2-4, wherein assigning the backup and / or restore operation comprises assigning the backup and / or restore operation to one or more nodes in one or more racks.

6. The method according to any of claims 1-5, wherein the restore operation comprises performing a preemptive restore operation.

7. The method according to claim 6, wherein the preemptive restore operation comprises iteratively, until all tables in the keyspace are restored:downloading a first table, or a first part of the first table, associated with the keyspace, andstreaming the first table, or the first part of the first table, to the selected datacenter (110) according to the assignment,while streaming the first table, or the first part of the first table, downloading a second table, or a second part of the first table, associated with the keyspace, and responsive to the streaming of the first table, or first part of the first table, being completed, streaming the second table, or second part of the first table, to the selected datacenter (110) according to the assignment.

8. The method according to claim 7, wherein prior to downloading a table, evaluating a size of the table, and upon determining that the size of the table is larger than a threshold, slicing the table into at least two parts, which parts are smaller than, or equal to, the threshold.

9. The method according to any of claims 1-8, wherein the backup operation comprises:determining, for each rack in the selected datacenter (110), whether all nodes in the rack are indicated as Up and Normal, UN, andresponsive to at least one node in a first rack being indicated as not being UN, selecting a second rack comprising only nodes indicated as UN.

10. The method according to any of claims 1-9, wherein the backup operation comprises applying a backup filter, which filter excludes temporary data from being backed up.

11. The method according to any one out of claims 1-10, wherein the keyspace is associated with a configuration, which configuration indicates any one or more out of:one or more datacenters the keyspace is replicated to, anda datacenter associated with the keyspace to be backed up.

12. The method according to any of claims 1-11 , wherein the datacenter is selected from the configuration associated with the keyspace.

13. A computer program (33) comprising instructions, which when executed by a processor (31), causes the processor (31) to perform actions according to any of the claims 1-12.

14. A carrier (34) comprising the computer program (33) of claim 13, wherein the carrier (34) is one of an electronic signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal, or a computer-readable storage medium.

15. A backup and / or restore controller (101) configured to handle de-duplication in a leaderless distributed database system (100) comprising a set of datacenters, wherein no datacenter (110) in the set of datacenters is configured as a dedicated backup datacenter, the backup and / or restore controller (101) further being configured to:select a datacenter (110) from the set of datacenters in the distributed database system (100) for a keyspace associated with a backup or restore operation,obtain a replication factor for the keyspace,based on a replication factor for the keyspace, assign the backup and / or restore operation to one or more nodes in the selected datacenter (110)perform the backup and / or restore operation.

16. The backup and / or restore controller (101) according to claim 15, wherein each datacenter comprises a set of nodes, and where the set of nodes are grouped into one or more racks.

17. The backup and / or restore controller (101) according to claim 16, wherein each rack comprises a replica of data associated with the keyspace, and wherein the data is distributed between the nodes in the rack.

18. The backup and / or restore controller (101) according to claim 16, wherein a rack comprises a replica of data associated with the keyspace, and wherein the data is distributed between the nodes in the rack.

19. The backup and / or restore controller (101) according to any of claims 16-18, wherein assigning the backup and / or restore operation comprises assigning the backup and / or restore operation to one or more nodes in one or more racks.

20. The backup and / or restore controller (101) according to any of claims 15-19, wherein the restore operation comprises performing a preemptive restore operation.

21. The backup and / or restore controller (101) according to claim 20, wherein the preemptive restore operation comprises iteratively, until all tables in the keyspace are restored:downloading a first table, or a first part of the first table, associated with the keyspace, andstreaming the first table, or the first part of the first table, to the selected datacenter (110) according to the assignment,while streaming the first table, or the first part of the first table, downloading a second table, or a second part of the first table, associated with the keyspace, and responsive to the streaming of the first table, or first part of the first table, being completed, streaming the second table, or second part of the first table, to the selected datacenter (110) according to the assignment.

22. The backup and / or restore controller (101) according to claim 21 , wherein prior to downloading a table, evaluating a size of the table, and upon determining that the size of the table is larger than a threshold, slicing the table into at least two parts, which parts are smaller than, or equal to, the threshold.

23. The backup and / or restore controller (101) according to any of claims 15-22, wherein the backup operation comprises:determining, for each rack in the selected datacenter (110), whether all nodes in the rack are indicated as Up and Normal, UN, andresponsive to at least one node in a first rack being indicated as not being UN, selecting a second rack comprising only nodes indicated as UN.

24. The backup and / or restore controller (101) according to any of claims 15-23, wherein the backup operation comprises applying a backup filter, which filter excludes temporary data from being backed up.

25. The backup and / or restore controller (101) according to any one out of claims 15-24, wherein the keyspace is associated with a configuration, which configuration indicates any one or more out of:one or more datacenters the keyspace is replicated to, anda datacenter associated with the keyspace to be backed up.

26. The backup and / or restore controller (101) according to any of claims 15-25, wherein the datacenter is selected from the configuration associated with the keyspace.

Images