Information processing device and information processing method

Abstract

This server (information processing device) comprises: a first rendering unit that performs rendering by three-dimensional computer graphics (3DCG) for an own assigned area assigned to the first rendering unit, in accordance with predetermined area allocation for a plurality of areas obtained by dividing a virtual space; a second rendering unit that performs rendering for a peripheral area surrounding the own assigned area at least on the basis of pre-generated 3D Gaussian splatting (3DGS) background data; and a generation unit that generates a three-dimensional view on the basis of a rendering result for the own assigned area by the first rendering unit and a rendering result for the peripheral area by the second rendering unit.

Description

Information Processing Apparatus and Information Processing Method

[0001] The present disclosure relates to an information processing apparatus and an information processing method.

[0002] Technologies related to a virtual space display system that displays a virtual space using three-dimensional computer graphics (3D CG) have been conventionally known (see Patent Document 1 below). In recent years, a technology for a large-scale multi-user connection infrastructure (such as MetaMe (registered trademark)) has been proposed, in which a single large virtual space is divided and each divided virtual space is rendered by a plurality of servers, enabling a very large number of users (for example, tens of thousands of users) to connect to the large virtual space. In this technology, the server connected to a certain user switches according to the area where the user is located in the virtual space. In the future, such a large-scale multi-user connection infrastructure is expected to spread widely.

[0003] Japanese Patent Application Laid-Open No. 11-306393

[0004] However, in the above technology, only a fixed camera is assumed, and for a three-dimensional view representing an area rendered by a server other than the server connected to a certain user (connection server) (that is, an external area other than the area where the user is located), the connection server cannot render the background of the external area and moving objects (other users, dynamic objects, etc.) existing in the external area. Therefore, for example, in a situation where a large number of users move across areas in the virtual space, it is assumed that the number of users displayed in the view of the external area suddenly decreases or increases at the timing when the servers connected to a large number of users switch, and there is a possibility that the sense of immersion felt by the users may decrease.

[0005] Therefore, an object of the present disclosure is to enable a user connected to a certain server to view a view of an external area with a free viewpoint (a camera that can be arbitrarily operated).

[0006] The information processing device according to this disclosure includes: a first rendering unit that renders its assigned area using three-dimensional computer graphics (3DCG) according to a predetermined area division for a plurality of areas obtained by dividing a virtual space; a second rendering unit that renders the area surrounding the assigned area based on at least pre-generated 3D Gaussian splatting (3DGS) background data; and a generation unit that generates a three-dimensional view based on the rendering result of the assigned area by the first rendering unit and the rendering result of the surrounding area by the second rendering unit.

[0007] According to this disclosure, a user connected to a server can view an external area from a free viewpoint (a camera that can be operated arbitrarily). This prevents situations where, for example, many users move across areas within a virtual space, the number of users displayed in the view of the external area suddenly decreases or increases when the server connected to those many users switches, thus preventing a decrease in the sense of immersion experienced by the user.

[0008] This is a diagram of the entire system configuration, including multiple servers (information processing devices). (a) is a diagram showing an example where only areas adjacent to the self-assigned area are considered surrounding areas, and (b) is a diagram showing an example where, in addition to adjacent areas, areas that exist with one adjacent area in between from the self-assigned area are also considered surrounding areas. This is a flowchart of the three-dimensional view generation process. This is a flowchart of the process executed when a request for additional 3DGS data is received from another server. This is a diagram showing an example of a three-dimensional view when rendering is performed on the surrounding area based on both 3DGS background data and additional 3DGS data. This is a diagram showing an example of a three-dimensional view when rendering results for the surrounding area cannot be obtained. This is a diagram showing an example of a three-dimensional view when rendering is performed on the surrounding area based only on 3DGS background data. This is a diagram showing an example of the hardware configuration of an information processing device.

[0009] Hereinafter, an embodiment of the information processing apparatus and information processing method relating to this disclosure will be described with reference to the drawings.

[0010] Figure 1 shows an overall configuration diagram of a system 1 comprising multiple servers 10, which are information processing devices according to this disclosure. As shown in Figure 1, each of the multiple servers 10 is equipped with a virtual space control unit 10A, and the virtual space control unit 10A has the function of constructing and controlling a common virtual space (for example, MetaMe®, metaverse, etc.) in cooperation with the virtual space control units 10A of other servers 10. The constructed virtual space is divided into multiple areas in advance, and one server 10 is in charge of one area according to a predetermined area division. That is, each server 10 communicates with the terminal 20 of a user (also called a "player," but hereinafter referred to as "user") located in the area it is in charge of in the virtual space (hereinafter referred to as its "personal area"), provides the terminal 20 with a three-dimensional view, and enables the user located in its personal area to view the three-dimensional view.

[0011] Each server 10 has a virtual space control unit 10A that, in addition to the conventional function of constructing and controlling the common virtual space described above, includes a first rendering unit 11, a second rendering unit 12, a generation unit 13, and an additional data provision unit 14 in order to realize the functions related to this disclosure described later. The functions of each unit will be outlined below.

[0012] The first rendering unit 11 is a functional unit that renders its own server 10's assigned area using three-dimensional computer graphics (3DCG).

[0013] The second rendering unit 12 is a functional unit that renders the area surrounding its own assigned area, at least based on pre-generated 3D Gaussian Splatting (3DGS) background data. 3D Gaussian Splatting (3DGS) is a technology for generating three-dimensional views, and one of the features of this disclosure is its effective use of 3DGS. In this embodiment, the second rendering unit 12 requests additional 3DGS data, including user information present in the surrounding area, from other servers 10 that share the surrounding area, and renders the surrounding area based on both the additional 3DGS data obtained from the other servers 10 and the above-mentioned 3DGS background data. The second rendering unit 12 also stores and manages pre-generated 3DGS background data in its built-in 3DGS background database (3DGS background DB) 12A.

[0014] The "surrounding area" may consist only of the area adjacent to the central self-assigned area (the area enclosed by the thick line) in Figure 2(a), as shown in Figure 2(a), or it may consist of the adjacent area plus areas that are separated from the central self-assigned area (the area enclosed by the thick line) in Figure 2(b) by a predetermined number of areas (one in Figure 2(b) as an example). The scope of what is included in such a "surrounding area" is predetermined.

[0015] The generation unit 13 is a functional unit that generates a three-dimensional view visible to a user located in its assigned area, based on the rendering results of its assigned area by the first rendering unit 11 and the rendering results of the surrounding area by the second rendering unit 12.

[0016] The additional data provision unit 14 is a functional unit that, upon receiving a request for 3DGS additional data from another server 10, generates 3DGS additional data including user information located in its own assigned area and provides the generated 3DGS additional data to the requesting server.

[0017] Each of the multiple servers 10 is equipped with a virtual space control unit 10A configured as described above. In one server 10, the first rendering unit 11 renders its own assigned area (the central area enclosed by thick lines in Figures 2(a) and (b)) based on 3DCG, while the second rendering unit 12 renders the surrounding areas (areas other than the central area in Figures 2(a) and (b)) based on 3DGS. The generation unit 13 generates a three-dimensional view visible to a user located in its assigned area based on the rendering result of its assigned area and the rendering results of multiple surrounding areas. As a result, a user connected to a server (i.e., a user located in its assigned area) can view not only the three-dimensional view of its assigned area but also the three-dimensional views of the surrounding areas from a free viewpoint.

[0018] The following describes the processes executed on server 10 using the flowcharts in Figures 3 and 4.

[0019] As shown in Figure 3, in one server 10, the first rendering unit 11 performs 3D CG rendering for its assigned area (step S1), and the second rendering unit 12 requests additional 3D CG data, including user information present in the surrounding area, from other servers 10 that are responsible for the surrounding area (step S2).

[0020] Here, using Figure 4, we will explain the process executed by another server that receives a request for 3DGS additional data. The process in Figure 4 is triggered by the receipt of a request for 3DGS additional data from another server. The additional data provision unit 14 of the server that received the request generates 3DGS additional data including user information that exists in its own assigned area (step S11), and provides (transmits) the generated 3DGS additional data to the requesting server 10 (step S12).

[0021] Returning to Figure 3, at server 10, the source of the 3DGS additional data request, the second rendering unit 12 acquires (receives) the 3DGS additional data provided by another server 10 (step S3), and acquires 3DGS background data from its built-in 3DGS background DB 12A (step S4). Then, the second rendering unit 12 performs rendering targeting the surrounding area based on both the 3DGS background data and the 3DGS additional data of the surrounding area (step S5).

[0022] The rendering results in steps S1 and S5 above are sent to the generation unit 13. Based on the rendering results of its own assigned area by the first rendering unit 11 and the rendering results of the surrounding area by the second rendering unit 12, the generation unit 13 generates three-dimensional view data visible to a user located in its assigned area (step S6), and outputs (transmits) the generated three-dimensional view data to the user's terminal 20 located in its assigned area (step S7).

[0023] In this way, the server 10 generates and outputs three-dimensional view data that includes not only its own assigned area but also the surrounding area. As a result, users located in their assigned area can view a three-dimensional view of both their assigned area and the surrounding area from a free viewpoint (a camera that can be operated arbitrarily). Conventionally, it was not possible to generate three-dimensional view data for the surrounding area, so only a three-dimensional view of the assigned area alone could be viewed, as shown in Figure 6, for example. In contrast, in this embodiment, it is possible to view a three-dimensional view of both the assigned area and the surrounding area, as shown in Figure 5, for example.

[0024] This prevents situations where, for example, a large number of users move across areas within a virtual space, the number of users displayed in the view of the surrounding area suddenly decreases or increases when the server connected to those users switches, thus preventing a decrease in the immersive experience for the user.

[0025] As a variation, the second rendering unit 12 may render based solely on 3DGS background data for the surrounding area without requesting or acquiring additional 3DGS data from other servers 10. In this case, for example, as shown in Figure 7, a three-dimensional view of the user's assigned area and the surrounding area including only the background can be viewed. Although the three-dimensional view in Figure 7 does not include moving objects in the surrounding area (user, car, etc.), the background of the surrounding area connected to the background of the user's assigned area can be viewed very naturally, thus reducing the sense of incongruity experienced by the user and maintaining a certain degree of immersion.

[0026] Furthermore, in the configuration shown in Figure 1, it is not mandatory for the second rendering unit 12 to store and manage pre-generated 3DGS background data in its built-in 3DGS background DB 12A. The 3DGS background data may be stored and managed by an external device other than the server 10, and the second rendering unit 12 may obtain the 3DGS background data from the external device by requesting it from the external device when needed.

[0027] The gist of this disclosure is found in the following [1] to [8]. [1] An information processing apparatus comprising: a first rendering unit that renders its assigned area using three-dimensional computer graphics (3DCG) according to a predetermined area division for a plurality of areas obtained by dividing a virtual space; a second rendering unit that renders the area surrounding the assigned area based on at least pre-generated 3D Gaussian splatting (3DGS) background data; and a generation unit that generates a three-dimensional view based on the rendering result of the assigned area by the first rendering unit and the rendering result of the surrounding area by the second rendering unit. [2] The information processing apparatus according to [1], wherein the second rendering unit further renders the surrounding area based on additional 3DGS data, including user information present in the surrounding area, obtained from other devices that share the surrounding area, in addition to the 3DGS background data. [3] The information processing apparatus according to [1] or [2], wherein the surrounding area includes at least an area adjacent to the self-assigned area. [4] The information processing apparatus according to [3], wherein the surrounding area includes, in addition to the adjacent area, an area that exists with a predetermined number of areas in between from the self-assigned area. [5] An information processing method comprising: the steps of: rendering the self-assigned area, which is assigned to the information processing apparatus, using three-dimensional computer graphics (3DCG) according to a predetermined area assignment for a plurality of areas obtained by dividing a virtual space; rendering the surrounding area of ​​the self-assigned area based at least on pre-generated 3D Gaussian splatting (3DGS) background data; and generating a three-dimensional view based on the rendering result of the self-assigned area and the rendering result of the surrounding area.

[0028] [Explanation of terms, explanation of hardware configuration (Figure 8), etc.]

[0029] The block diagrams used in the description of the above embodiments show functional units. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or it may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired or wireless connections). A functional block may also be realized by combining software with the one or more of the above devices.

[0030] Functions include, but are not limited to, judgment, decision, judgment, calculation, calculation, processing, derivation, investigation, exploration, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, assumption, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating (mapping), and assigning. For example, a functional block (configuration part) that enables transmission is called a transmitting unit or transmitter. As mentioned above, the method of implementation is not particularly limited.

[0031] For example, the information processing device in one embodiment of the present disclosure may function as a computer that performs the processing of the present disclosure. Figure 8 is a diagram showing an example of the hardware configuration of the information processing device 10 according to one embodiment of the present disclosure. The information processing device 10 described above may be physically configured as a computer device including a processor 1001, memory 1002, storage 1003, communication device 1004, input device 1005, output device 1006, bus 1007, etc.

[0032] In the following explanation, the term "device" can be replaced with "circuit," "device," "unit," etc. The hardware configuration of the information processing device 10 may include one or more of the devices shown in the figure, or it may be configured to omit some of the devices.

[0033] Each function in the information processing device 10 is realized by loading predetermined software (programs) onto hardware such as the processor 1001 and memory 1002, which allows the processor 1001 to perform calculations, control communication by the communication device 1004, and control at least one of data reading and writing in the memory 1002 and storage 1003.

[0034] The processor 1001 controls the entire computer, for example, by running an operating system. The processor 1001 may be composed of a central processing unit (CPU) that includes interfaces with peripheral devices, control units, arithmetic units, registers, etc.

[0035] Furthermore, the processor 1001 reads programs (program code), software modules, data, etc., from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes accordingly. The program used is one that causes the computer to execute at least a part of the operations described in the above embodiment. Although it has been explained that the various processes are executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may also be transmitted from a network via a telecommunications line.

[0036] The memory 1002 is a computer-readable recording medium and may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. The memory 1002 may also be called a register, cache, main memory, etc. The memory 1002 can store executable programs (program code), software modules, etc., for carrying out a wireless communication method according to one embodiment of the present disclosure.

[0037] The storage 1003 is a computer-readable recording medium and may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disc, a digital multipurpose disc, a Blu-ray® disc), a smart card, flash memory (e.g., a card, a stick, a key drive), a floppy® disk, a magnetic strip, etc. The storage 1003 may also be called an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, server, or other suitable medium including at least one of the memory 1002 and the storage 1003.

[0038] The communication device 1004 is hardware (transceiver / receiver device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as a network device, network controller, network card, communication module, etc. The communication device 1004 may be configured to include, for example, a high-frequency switch, duplexer, filter, frequency synthesizer, etc., in order to implement at least one of frequency division duplex (FDD) and time division duplex (TDD).

[0039] The input device 1005 is an input device that accepts input from an external source (e.g., a keyboard, mouse, microphone, switch, button, sensor, etc.). The output device 1006 is an output device that outputs to an external source (e.g., a display, speaker, LED lamp, etc.). The input device 1005 and the output device 1006 may be configured as an integrated unit (e.g., a touch panel).

[0040] Furthermore, each device, such as the processor 1001 and memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or different buses may be configured for each device.

[0041] Furthermore, the information processing device 10 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array), and some or all of each functional block may be realized by such hardware. For example, the processor 1001 may be implemented using at least one of these hardware components.

[0042] The notification of information is not limited to the embodiments described herein and may be carried out by other means. For example, the notification of information may be carried out by physical layer signaling (e.g., DCI (Downlink Control Information), UCI (Uplink Control Information)), upper layer signaling (e.g., RRC (Radio Resource Control) signaling, MAC (Medium Access Control) signaling, broadcast information (MIB (Master Information Block), SIB (System Information Block))), other signals, or combinations thereof. RRC signaling may also be called RRC messages, and may be, for example, RRC Connection Setup messages, RRC Connection Reconfiguration messages, etc.

[0043] Each aspect / embodiment described in the present disclosure may be applied to at least one of systems that utilize LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (xG (x is, for example, an integer or a decimal)), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), and other appropriate systems, as well as next-generation systems extended, modified, generated, and defined based on these. Further, a combination of multiple systems (for example, a combination of at least one of LTE and LTE-A and 5G, etc.) may be applied.

[0044] The processing procedures, sequences, flowcharts, etc. of each aspect / embodiment described in the present disclosure may be reordered as long as there is no contradiction. For example, regarding the methods described in the present disclosure, the elements of various steps are presented using an exemplary order and are not limited to the specific order presented.

[0045] The input / output information, etc. may be stored in a specific location (for example, a memory) or may be managed using a management table. The input / output information, etc. may be overwritten, updated, or appended. The output information, etc. may be deleted. The input information, etc. may be transmitted to other devices.

[0046] The determination may be made based on a value represented by 1 bit (either 0 or 1), a boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0047] Each aspect / embodiment described in the present disclosure may be used alone, in combination, or switched and used during execution. Also, the notification of predetermined information (e.g., notification of "being X") is not limited to being explicitly performed, and may be performed implicitly (e.g., by not performing the notification of the predetermined information).

[0048] As described above in detail regarding the present disclosure, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented as modifications and variations without departing from the spirit and scope of the present disclosure defined by the claims. Therefore, the description of the present disclosure is for the purpose of illustrative explanation and has no restrictive meaning for the present disclosure.

[0049] Software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether it is called software, firmware, middleware, microcode, a hardware description language, or another name.

[0050] Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technology (such as infrared or microwave), then at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0051] The information, signals, etc. described in this disclosure may be represented using any of the various different techniques. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0052] In addition, terms used in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of the channel and symbol may be a signal (signaling). Also, a signal may be a message. Furthermore, a component carrier (CC) may be called a carrier frequency, cell, frequency carrier, etc.

[0053] The terms “system” and “network” as used in this disclosure are interchangeable.

[0054] Furthermore, the information, parameters, etc., described in this disclosure may be expressed using absolute values, relative values ​​from a given value, or other corresponding information. For example, wireless resources may be indicated by an index.

[0055] The names used for the parameters described above are not restrictive in any way. Furthermore, the formulas and other expressions using these parameters may differ from those expressly disclosed in this disclosure. Various channels (e.g., PUCCH, PDCCH, etc.) and information elements can be identified by any suitable name, and therefore, the various names assigned to these various channels and information elements are not restrictive in any way.

[0056] As used in this disclosure, the terms “determining” and “determining” may encompass a wide variety of actions. “Determining” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, or inquiring (e.g., searching in a table, database, or other data structure), or ascertaining. “Determining” may also include receiving (e.g., receiving information), transmitting (e.g., sending information), inputting, outputting, or accessing (e.g., accessing data in memory). Furthermore, "judgment" and "decision" can include considering something as having been "judged" or "decided" after resolving, selecting, choosing, establishing, comparing, etc. In other words, "judgment" and "decision" can include considering something as having been "judged" or "decided" after some action. Also, "judgment (decision)" can be reinterpreted as "assuming," "expecting," or "considering."

[0057] In this disclosure, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on."

[0058] Any reference to elements using the designations “first,” “second,” etc., as used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Accordingly, references to the first and second elements do not imply that only two elements may be employed, or that the first element must precede the second element in any way.

[0059] Where the terms “include,” “including,” and variations thereof are used in this disclosure, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to mean exclusive OR.

[0060] In this disclosure, if articles are added through translation, such as a, an, and the in English, this disclosure may include the fact that the noun following these articles is plural.

[0061] In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combine" may be interpreted similarly to "different."

[0062] 1...System, 10...Server (information processing device), 10A...Virtual space control unit, 11...First rendering unit, 12...Second rendering unit, 13...Generation unit, 14...Additional data provision unit, 20...Terminal, 1001...Processor, 1002...Memory, 1003...Storage, 1004...Communication device, 1005...Input device, 1006...Output device, 1007...Bus.

Claims

1. An information processing device comprising: a first rendering unit that renders its assigned area using three-dimensional computer graphics (3DCG) according to a predetermined area division for a plurality of areas obtained by dividing a virtual space; a second rendering unit that renders the area surrounding the assigned area based on at least pre-generated 3D Gaussian splatting (3DGS) background data; and a generation unit that generates a three-dimensional view based on the rendering result of the assigned area by the first rendering unit and the rendering result of the surrounding area by the second rendering unit.

2. The information processing apparatus according to claim 1, wherein the second rendering unit renders the surrounding area based on additional 3DGS data, including user information present in the surrounding area, obtained from other devices responsible for the surrounding area, in addition to the 3DGS background data.

3. The information processing apparatus according to claim 1, wherein the surrounding area includes at least an area adjacent to the self-assigned area.

4. The information processing apparatus according to claim 3, wherein the surrounding area includes, in addition to the adjacent area, an area that is separated from the self-assigned area by a predetermined number of other areas.

5. An information processing method comprising: a step of rendering a self-assigned area, which is the area to which the information processing device is responsible, using three-dimensional computer graphics (3DCG) according to a predetermined area assignment for a plurality of areas obtained by dividing a virtual space; a step of rendering a surrounding area of ​​the self-assigned area, based at least on pre-generated 3D Gaussian splatting (3DGS) background data; and a step of generating a three-dimensional view based on the rendering result of the self-assigned area and the rendering result of the surrounding area.

Images