Cloud native-based real-time XR cloud computing system in multi-cloud environment

Abstract

This XR cloud computing system for providing a real-time XR content service by utilizing a microservice-based cloud native in a multi-cloud environment comprises: a computing virtualization unit for supporting the multi-cloud environment, virtualizing computing resources, and managing microservices; a content loading unit for loading large-capacity XR content; a content rendering unit for rendering the XR content in the cloud environment; a content encoding unit for performing an optimization operation in order to transmit the rendered XR content with ultra-low latency; a content interaction unit for transmitting the XR content and allowing interaction with an XR device of a user; a cache unit for accelerating processing speeds of the modules; and an XR device SDK unit for decoding the received XR content and rendering the decoded XR content on a device of the user.

Description

Cloud-native based real-time XR cloud computing system in a multi-cloud environment

[0001] The present invention relates to an XR cloud computing system for providing real-time XR content services by utilizing cloud-native based on a microservices architecture in a multi-cloud environment. Specifically, it relates to a computing system for providing real-time XR content services by offloading the processing of large-capacity XR content, which is difficult to process on XR devices, to a cloud-native environment and transmitting the processing results with ultra-low latency.

[0002] XR Cloud is a cloud computing-based service and infrastructure technology that supports and provides extended reality technologies such as Augmented Reality (AR) and Virtual Reality (VR). Through this, it enables users to have rich interactions between the real and virtual environments and provides a technical environment that delivers high-quality XR content in real time via various XR devices.

[0003] Gartner’s 2023 Top 10 Strategic Technology Trends predicted that a combination of XR cloud and digital twins would be used in metaverse-based projects worldwide by 2027. Gartner predicts that XR cloud will have a very high impact as it will change the way people interact with the world around them, and reports that XR cloud will provide a digital abstraction layer for people, places, and things, securing a foothold across business and consumer applications and impacting all industries regardless of region.

[0004] As such, the application of cloud computing technology is necessary to overcome the technical limitations (processing capabilities of XR devices, network latency) of Extended Reality (XR), which is emerging as a new future market. If the Motion-to-Photon (MTP) response time to user input or actions exceeds 20ms, a phenomenon known as VR Sickness occurs, severely degrading the Quality of Experience (QoE). Since XR content is generally much larger than standard content (typically over 12K), failure to utilize cloud computing technology that takes these characteristics into account inevitably leads to increased MTP latency due to transmission issues with large-capacity content.

[0005] For example, NVIDIA offers a service through its game cloud platform, GeForce Now, that allows users to enjoy high-spec games regardless of the performance of their devices. However, the service has not achieved great success as it struggles to provide high-quality content at the level users desire due to high transmission rates during the process of streaming large-capacity media or 3D game content in real time.

[0006] Therefore, the objective of the present invention is to construct a computing system capable of providing high-quality XR content services by utilizing microservice-based cloud-native in a multi-cloud environment to offload XR content processing, which requires significant computing resources, to the cloud environment for processing, and to transmit it to user XR devices in a lightweight manner for ultra-low latency.

[0007] An XR cloud computing system for providing real-time XR content services using microservice-based cloud native in a multi-cloud environment according to the present invention for achieving the above objective comprises: a computing virtualization unit that supports a multi-cloud environment, virtualizes computing resources, and controls microservices; a content loading unit that loads large-capacity XR content; a content rendering unit that renders XR content in a cloud environment; a content encoding unit that performs optimization work to transmit rendered XR content with ultra-low latency; a content interaction unit for transmitting XR content and interacting with a user's XR device; a cache unit for accelerating the processing speed of the above modules; and an XR device SDK unit for decoding transmitted XR content and rendering it to a user device.

[0008] Here, the computing virtualization unit can virtualize computing for processing XR content and control XR content processing microservices in a multi-cloud environment using cloud environments of various cloud providers such as AWS, Azure, and Google.

[0009] And the content loading department can load content to service broadcasts, 3D games, etc. as XR content.

[0010] Here, the content rendering unit may render the XR content loaded by the content loading unit on the cloud first to undergo an encoding process, rather than transmitting it directly to the user's XR device.

[0011] And the content encoding unit can encode XR content rendered by the content rendering unit in real time for ultra-low latency transmission.

[0012] Here, the content interaction unit can transmit encoded XR content to the user's XR device and also receive the user's interaction from the user's XR device and transmit it to the XR cloud system.

[0013] And the cache unit can provide in-memory caching for various data to improve the processing speed of all modules running on the computing virtualization unit, such as content loading, rendering, encoding, and interaction.

[0014] Here, the XR device SDK part can decode the encoded and transmitted XR content, render it on the user's XR device, and transmit various interactions of the user to the XR cloud computing system.

[0015] According to the present invention, computing virtualization of CPUs, GPUs, etc., capable of processing XR content by integrating a multi-cloud environment through a computer virtualization unit is provided, and efficient utilization of computing resources is possible through Auto Scaling of microservices as well as computing environments.

[0016] In addition, by loading large-capacity multi-view XR content through the content loading section and pre-warming the loaded content in the cache, the content rendering section, which is a subsequent processing module, can quickly access the content.

[0017] In addition, through the content rendering unit, large-capacity XR content is rendered in cloud computing rather than directly on XR devices with low computing power, thereby enabling high-quality XR content services.

[0018] In addition, the content encoding section has the effect of optimizing large-capacity XR content rendered in cloud computing for transmission over a network.

[0019] In addition, by transmitting encoded XR content to the user's XR device through the content interaction unit and also receiving the user's interaction in real time and transmitting it to cloud computing, it is possible to provide an ultra-low latency interaction service.

[0020] In addition, through the cache section, multi-view XR content, rendering results, encoding results, etc. are cached in an In-Memory storage, which has the effect of enabling ultra-low latency processing.

[0021] In addition, XR content encoded and transmitted through the XR device SDK is decoded and rendered on the user's XR device, and the user's interaction is transmitted to cloud computing in real time, which has the effect of enabling seamless, high-quality XR content services.

[0022] FIG. 1 is an overall block diagram of an XR cloud computing system for providing real-time XR content services by utilizing microservice-based cloud native in a multi-cloud environment according to the present invention.

[0023] Figure 2 is a block diagram of the computing virtualization unit.

[0024] Fig. 3 is a block diagram of the content loading section.

[0025] Fig. 4 is a block diagram of the content rendering section.

[0026] Figure 5 is a block diagram of the content encoding section.

[0027] Fig. 6 is a block diagram of the content interaction section.

[0028] Fig. 7 is a block diagram of the cache section.

[0029] Fig. 8 is a block diagram of the XR device SDK section.

[0030] Hereinafter, an XR cloud computing system (1) for providing real-time XR content services using microservice-based cloud native in a multi-cloud environment according to a preferred embodiment of the present invention will be described in detail with reference to the attached drawings.

[0031] FIG. 1 is an overall block diagram of an XR cloud computing system (1) for providing real-time XR content services using microservice-based cloud native in a multi-cloud environment according to the present invention, FIG. 2 is a block diagram of a computing virtualization unit (10), FIG. 3 is a block diagram of a content loading unit (20), FIG. 4 is a block diagram of a content rendering unit (30), FIG. 5 is a block diagram of a content encoding unit (40), FIG. 6 is a block diagram of a content interaction unit (50), FIG. 7 is a block diagram of a cache unit (60), and FIG. 8 is a block diagram of an XR device SDK unit (70).

[0032] Referring to FIGS. 1 to 7, the configuration of an XR cloud computing system (1) for providing real-time XR content services using microservice-based cloud native in a multi-cloud environment is described.

[0033] An XR cloud computing system (1) for providing real-time XR content services using microservice-based cloud native in a multi-cloud environment includes a computing virtualization unit (10), a content loading unit (20), a content rendering unit (30), a content encoding unit (40), a content interaction unit (50), a cache unit (60), and an XR device SDK unit (70).

[0034] The virtualization unit (10) supports a multi-cloud environment, virtualizes computing resources, and manages microservices.

[0035] The content loading section (20) can load large-capacity XR content.

[0036] The content rendering unit (30) can render XR content in a cloud environment.

[0037] The content encoding unit (40) can perform optimization work to transmit rendered XR content with ultra-low latency.

[0038] The content interaction unit (50) can transmit XR content and interact with the user's XR device.

[0039] The cache section (60) can accelerate the processing speed of the modules.

[0040] The XR device SDK part (70) can decode the transmitted XR content and render it to the user device.

[0041]

[0042] As shown in Fig. 1, computing is virtualized on a multi-cloud, and on the virtualized computing, a content loading unit, a content rendering unit, a content encoding unit, a content interaction unit, and a cache unit each operate as microservices. Separately, the XR device SDK unit operates on the user's XR device.

[0043] As shown in Fig. 2, the computing virtualization unit first supports a multi-cloud environment to integrate computing resources such as CPUs and GPUs. On top of these integrated computing resources, the content loading unit, content rendering unit, content encoding unit, content interaction unit, and cache unit for XR content processing are each deployed as microservices. The computing virtualization unit is responsible for managing these microservices; when a specific microservice becomes overloaded, it instantaneously deploys more of that microservice within the available range of current computing resources to ensure that the bottleneck processing is processed quickly. If the surged processing ends and there are many idle microservices among the additionally deployed microservices, the deployed microservices are reclaimed to secure computing resources. Furthermore, if there are no more computing resources available to deploy microservices, additional computing resources are secured through an Auto Scaling policy.

[0044] As shown in Fig. 3, the content loading unit loads XR content such as broadcasts and 3D games. Generally, since broadcast content with a fixed viewpoint is insufficient for service on XR devices, multi-view video content captured simultaneously by multiple cameras is loaded in the case of broadcast content. Additionally, in the case of 3D games, since various viewpoint changes are possible within the game, multi-view game content is loaded in the same way as broadcast content. Because the loaded multi-view content is large in size, it is stored in a cache so that it can be quickly utilized in subsequent processing.

[0045] As shown in Fig. 4, the content rendering unit renders multi-view content in a cloud environment. Transmitting large-capacity multi-view content as is to render it on an XR device would inevitably result in a decline in content quality due to the limited computing power of the device. Therefore, multi-view content is rendered on cloud computing, and optimized content is transmitted to the user's XR device. The content rendering unit starts operating when a cache save completion signal is received from the content loading unit, and loads and renders the multi-view content from the cache. At this time, each multi-view content is rendered simultaneously in each microservice.

[0046] As shown in Fig. 5, the content encoding unit consists of a multi-view separator, a depth information extractor / estimator, a deduplication unit, a restoration information generator, and a data encoder. The content encoding unit operates simultaneously when multi-view content is rendered in the content rendering unit. The multi-view separator selects one of the rendered multi-view contents as the Main View and groups the remaining viewpoints into Additional Views. The depth information extractor / estimator generates depth map data by extracting depth information from each viewpoint image or estimating it using an artificial intelligence model. The deduplication unit utilizes the depth map data generated by the depth information extractor / estimator to remove parts from the Additional View that overlap with the Main View—that is, parts that can be safely removed—based on the Main View. It then cuts the non-overlapping parts from the Additional View into a Patch and appends them to the Main View. The restoration information generator constructs information on how to restore the original viewpoint view using the Main View and the Patch. The data encoder encodes the Main View, depth map data, Patch data, and restoration information, converts them into a form optimized for transmission over a network, and stores them in a cache.

[0047] As shown in FIG. 6, the content interaction unit consists of a bitstream generator / transmitter that transmits XR content and a user interaction receiver that receives user interactions from an XR device. The bitstream generator retrieves the encoding result stored in the cache from the content encoding unit and converts it into a bitstream for network transmission. The bitstream transmitter transmits the generated bitstream to the user terminal using a protocol designed for ultra-low latency transmission. The user interaction receiver receives the interaction between the user and the content sent from the user's XR device and transmits it to the content rendering unit. Based on the user interaction, the content rendering unit re-renders the corresponding multi-view content, and this content is re-encoded and transmitted to the user.

[0048] As shown in FIG. 7, the cache section consists of a data input / output API, a data reader, and an in-memory storage. The data input / output API provides APIs for storing data in the cache and loading the stored data. At this time, the API response speed must be O(1) or at least faster than O(logn) when the data size is n. The data reader determines whether the data being stored is binary data or string data, etc. The in-memory storage stores the data in a structure that can have the fastest response speed depending on the form of the read data.

[0049] As shown in FIG. 8, the XR device SDK section consists of a data decoder, a data renderer, and a user interaction transmitter. The data decoder decodes and restores the bitstream transmitted over the network to construct the original multi-view content. The constructed multi-view content is passed to the data renderer and rendered on the user's XR device. At this time, interactions between the user and the content are transmitted to the XR cloud computing system through the user interaction transmitter, and the XR cloud computing system processes the corresponding XR content with ultra-low latency processing to enable a seamless and immersive XR content service.

[0050] Due to the XR cloud computing system (1) for providing real-time XR content services by utilizing microservice-based cloud native in the above multi-cloud environment, computing virtualization such as CPUs and GPUs that can process XR content by integrating the multi-cloud environment through a computer virtualization unit is provided, and efficient utilization of computing resources is possible through Auto Scaling of microservices as well as computing environments.

[0051] In addition, a large volume of multi-view XR content is loaded through the content loading section, and the loaded content is pre-warmed in the cache, allowing the content to be accessed quickly in the content rendering section, which is a subsequent processing module.

[0052] In addition, through the content rendering unit, high-quality XR content services can be provided by rendering large-capacity XR content in cloud computing instead of rendering it directly on XR devices with low computing power.

[0053] In addition, the content encoding unit encodes large-capacity XR content rendered in cloud computing and optimizes it for transmission over a network.

[0054] In addition, an ultra-low latency interaction service is possible by transmitting encoded XR content to the user's XR device through the content interaction unit, and also by receiving the user's interaction in real time and transmitting it to cloud computing.

[0055] In addition, through the cache section, multi-view XR content, rendering results, encoding results, etc. are cached in an In-Memory storage, enabling ultra-low latency processing.

[0056] In addition, XR content encoded and transmitted through the XR device SDK is decoded and rendered on the user's XR device, and the user's interaction is also transmitted to cloud computing in real time, enabling seamless high-quality XR content services.

Claims

1. In an XR cloud computing system for providing real-time XR content services by utilizing microservice-based cloud-native in a multi-cloud environment, A computing virtualization unit that supports multi-cloud environments, virtualizes computing resources, and manages microservices; Content loading section for loading large-capacity XR content; Content rendering unit for rendering XR content in a cloud environment; A content encoding unit that performs optimization work for ultra-low latency transmission of rendered XR content; Content interaction section for transmitting XR content and interacting with the user's XR device; A cache unit for accelerating the processing speed of the above modules; and An XR cloud computing system for providing real-time XR content services utilizing microservice-based cloud-native in a multi-cloud environment, characterized by including an XR device SDK part for decoding transmitted XR content and rendering it on a user device.

2. In Paragraph 1, The above computing virtualization unit, An XR cloud computing system for providing real-time XR content services by utilizing microservice-based cloud-native in a multi-cloud environment, characterized by virtualizing computing for processing XR content and controlling XR content processing microservices in a multi-cloud environment using cloud environments of various cloud providers.

3. In Paragraph 1, The above content loading section is, An XR cloud computing system for providing real-time XR content services by utilizing microservice-based cloud-native in a multi-cloud environment characterized by loading content to be serviced as XR content including broadcasting and 3D games.

4. In Paragraph 1, The above content rendering unit, An XR cloud computing system for providing real-time XR content services by utilizing microservice-based cloud-native in a multi-cloud environment, characterized by rendering XR content loaded in the content loading section on the cloud first to undergo an encoding process, rather than transmitting the XR content as is to the user's XR device.

5. In Paragraph 1, The above content encoding unit, An XR cloud computing system for providing real-time XR content services by utilizing microservice-based cloud-native in a multi-cloud environment, characterized by real-time encoding of XR content rendered in a content rendering unit for ultra-low latency transmission.

6. In Paragraph 1, The above content interaction section is, An XR cloud computing system for providing real-time XR content services by utilizing microservice-based cloud-native in a multi-cloud environment, characterized by transmitting encoded XR content to a user's XR device and also receiving user interactions from the user's XR device and delivering them to an XR cloud system.

7. In Paragraph 1, An XR cloud computing system for providing real-time XR content services utilizing microservices-based cloud-native in a multi-cloud environment, characterized by a cache section that provides in-memory caching for various data to improve the processing speed of all modules operating on the computing virtualization section, such as content loading, rendering, encoding, and interaction.

8. In Paragraph 1, An XR cloud computing system for providing real-time XR content services utilizing microservice-based cloud-native in a multi-cloud environment, characterized by the above-mentioned XR device SDK part decoding encoded and transmitted XR content, rendering it on the user's XR device, and transmitting various user interactions to the XR cloud computing system.

Images