Information processing program, terminal, information processing device, and information processing system
The system addresses the challenge of rendering 3D data by generating and streaming rendering data with color and depth from a single viewpoint, enabling accurate 3D data reconstruction and playback on 3D displays.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- SOFTGEAR CO LTD
- Filing Date
- 2025-11-19
- Publication Date
- 2026-06-11
AI Technical Summary
Conventional information processing systems struggle to directly apply two-dimensional color and depth data streaming to game servers handling three-dimensional data, limiting their ability to reconstruct and render 3D data.
An information processing system that generates and streams rendering data comprising color and depth data from a single viewpoint, allowing terminals to reconstruct and render three-dimensional objects by processing received data, including a server that processes basic data into rendering data and distributes it, and terminals that receive, reconstruct, and render this data for 3D displays.
Enables the reconstruction and rendering of three-dimensional data on the receiving side, even when streaming, allowing for accurate 3D data playback on various 3D displays.
Smart Images
Figure JP2025040378_11062026_PF_FP_ABST
Abstract
Description
Information Processing Program, Terminal, Information Processing Apparatus, and Information Processing System
[0001] The present invention relates to an information processing program, a terminal, an information processing apparatus, and an information processing system.
[0002] As a conventional technique, an information processing apparatus that streams two-dimensional color data and depth data has been proposed (for example, see Patent Document 1).
[0003] The information processing apparatus disclosed in Patent Document 1 captures surface data regarding one or more objects from a plurality of capture devices physically arranged at viewpoints with different perspectives related to a landscape including one or more objects, and generates, for each capture device included in the plurality of capture devices, a color video data stream regarding 2D color data and a depth video data stream regarding depth data, packages the color video data stream and the depth video data stream into a transfer stream for each capture device included in the plurality of capture devices, and provides metadata and the transfer stream for streaming to a media player device.
[0004] Japanese Patent Translation of PCT International Publication No. 2020522926
[0005] The above-described conventional information processing apparatus generates two-dimensional color data and depth data using a plurality of capture devices, packages video streams for each capture device, and performs streaming to reproduce a virtual space in a media player device. However, since it captures the real world, there is a problem that it cannot be directly applied to a game server that handles three-dimensional data.
[0006] An object of the present invention is to provide an information processing program, a terminal, an information processing apparatus, and an information processing system that can restore three-dimensional data on the receiving side even when rendering and streaming three-dimensional data.
[0007] To achieve the above object, one aspect of the present invention provides the following information processing program, terminal, information processing apparatus, and information processing system.
[0008] [1] An information processing program that causes a computer to function as a receiving means for receiving color data representing the color of each pixel in an image and depth data representing the distance from the one viewpoint to the surface of the object, for one or more viewpoints, as rendering data generated by rendering basic data representing an object as a three-dimensional object, which is streamed, based on one viewpoint; and a reconstruction means for reconstructing the basic data based on the received rendering data for one or more viewpoints. [2] The information processing program according to [1] that further functions as a rendering means for rendering the reconstructed basic data according to a playback device. [3] The information processing program according to [1] that expresses the distance of each point in the depth direction from the surface of the object based on the one viewpoint. [4] The information processing program according to [1] that estimates the material of the object from the color data as the reconstruction means. [5] An information processing program that causes a computer to function as a processing means for generating color data representing the color of each pixel in an image and depth data representing the distance from the one viewpoint to the surface of the object, for one or more viewpoints, as rendering data generated by rendering basic data representing an object as a three-dimensional object, based on one viewpoint; and a distribution means for streaming the rendering data. [6] A terminal having receiving means for receiving color data representing the color of each pixel in an image and depth data representing the distance from the one viewpoint to the surface of the object, as rendering data generated by rendering basic data representing an object as a three-dimensional object based on one viewpoint, which is streamed, for one or more viewpoints, and reconstruction means for reconstructing basic data based on the received rendering data for one or more viewpoints. [7] An information processing device having processing means for generating color data representing the color of each pixel in an image and depth data representing the distance from the one viewpoint to the surface of the object, as rendering data generated by rendering basic data representing an object as a three-dimensional object based on one viewpoint, for one or more viewpoints, and distribution means for streaming the rendering data.[8] An information processing device having processing means for generating rendering data, which is generated by rendering basic data that represents an object as a three-dimensional object based on one viewpoint, including color data that represents the color of each pixel in the image and depth data that represents the distance from the one viewpoint to the surface of the object, for one or more viewpoints; and distribution means for streaming the rendering data; and a terminal having receiving means for receiving the streaming rendering data and reconstruction means for reconstructing basic data based on the received rendering data for one or more viewpoints.
[0009] According to the inventions of claims 1, 5, 6, 7, and 8, even when rendering and streaming three-dimensional data, the receiving side can reconstruct the three-dimensional data. According to the invention of claim 2, the reconstructed basic data can be rendered according to the playback device. According to the invention of claim 3, the distance of each point in the depth direction from the surface of the object can be expressed as depth data based on a single viewpoint. According to the invention of claim 4, the material of the object can be estimated from the color data.
[0010] Figure 1 is a schematic diagram showing an example of the configuration of an information processing system according to an embodiment. Figure 2 is a block diagram showing an example of the configuration of an information processing device according to an embodiment. Figure 3 is a block diagram showing an example of the configuration of a terminal according to an embodiment. Figure 4 is a flowchart for explaining an example of the operation of an information processing device. Figure 5A is a schematic diagram for explaining the operation of rendering and reconstruction of basic data. Figure 5B is a schematic diagram for explaining the operation of rendering and reconstruction of basic data. Figure 5C is a schematic diagram for explaining the operation of rendering and reconstruction of basic data. Figure 6A is a schematic diagram for explaining the operation of rendering and reconstruction of basic data according to a second embodiment. Figure 6B is a schematic diagram for explaining the operation of rendering and reconstruction of basic data according to a second embodiment. Figure 6C is a schematic diagram for explaining the operation of rendering and reconstruction of basic data according to a second embodiment. Figure 7A is a schematic diagram for explaining the operation of rendering and reconstruction of basic data according to a third embodiment. Figure 7B is a schematic diagram for explaining the operation of rendering and reconstruction of basic data according to a third embodiment. Figure 7C is a schematic diagram for explaining the operation of rendering and reconstruction of basic data according to a third embodiment.
[0011] [First Embodiment] (Configuration of Information Processing System) Figure 1 is a schematic diagram showing an example of the configuration of an information processing system according to the embodiment.
[0012] This information processing system consists of a server 1 and terminals 2a, 2b, 2c... connected to each other via a network 4. Terminals 2a, 2b, 2c... are operated by users 3a, 3b, 3c..., respectively. Terminals 2a, 2b, 2c... are connected to 3D displays 23a, 23b, 23c..., which act as playback devices. The 3D displays 23a, 23b, 23c... display and control the data so that users 3a, 3b, 3c... can view it in stereoscopically.
[0013] Server 1 is a server-type information processing device that operates in response to requests from terminals 2a, 2b, 2c, etc., and is equipped with electronic components such as a CPU (Central Processing Unit) and flash memory that have functions for processing information.
[0014] Terminals 2a, 2b, 2c, etc. are information processing devices such as PCs (Personal Computers) that are equipped with electronic components such as a CPU and flash memory that have functions for processing information.
[0015] The 3D displays 23a, 23b, 23c, etc. can employ various methods, such as a frame sequential method using dedicated shutter glasses, a polarized method using polarizing plate glasses, and a parallax barrier method that allows viewing with the naked eye. They may also be HMDs (Head Mount Displays). Furthermore, the 3D video files to be played can use various methods depending on the display method, such as MVC, side-by-side, and top-and-bottom formats.
[0016] Network 4 is a communication network capable of high-speed communication, such as a wired or wireless communication network like the Internet.
[0017] Server 1 is, for example, a game server that operates in response to requests from terminals 2a, 2b, 2c, etc., and a viewpoint corresponding to each terminal 2a, 2b, 2c, etc. is set in 3D space. Server 1 renders objects in the field of view from the viewpoint corresponding to terminals 2a, 2b, 2c, etc., and streams the rendering data. Terminals 2a, 2b, 2c, etc. each reconstruct 3D data based on the streamed rendering data, render the reconstructed 3D data, and output data corresponding to the format of 3D displays 23a, 23b, 23c, etc. Users 3a, 3b, 3c, etc. view the images displayed on the 3D displays 23a, 23b, 23c, etc., and experience 3D images.
[0018] (Configuration of the information processing device) Figure 2 is a block diagram showing an example of the configuration of the server 1 according to the embodiment.
[0019] Server 1 comprises a control unit 10, which consists of a CPU and the like, controls each part and executes various programs; a storage unit 11, which consists of a storage medium such as flash memory and stores information; and a communication unit 12, which communicates with the outside world via the network 4.
[0020] The control unit 10 functions as a basic data processing means 100, a data distribution means 101, etc., by executing a streaming program 110, which will be described later as an information processing program.
[0021] The basic data processing means 100 processes the basic data 111 as 3D data and generates rendering data 112 based on a certain viewpoint at a set frame rate frequency. The basic data 111 is, for example, polygon mesh data, but voxel data may be used as the basic data 111 and converted to polygon mesh data during rendering. The rendering data 112 has color data 113 for each pixel of the rendering result image and depth data 114 which is the distance from a certain viewpoint. The depth data 114 is generated by the basic data processing means 100 acquiring information from the depth buffer in memory used during rendering. The depth data 114 has the depth of the points corresponding to the pixels of the color data 113, but it may have different granularities. The depth data 114 may also include the coordinates of the viewpoint and the rotation angle of the camera.
[0022] The data distribution means 101 streams rendering data 112 (color data 113, depth data 114) for each frame in timeline order.
[0023] The memory unit 11 stores a streaming program 110 that causes the control unit 10 to operate as each of the means 100101 described above, basic data 111, rendering data 112 (color data 113, depth data 114), etc.
[0024] (Terminal Configuration) Figure 3 is a block diagram showing an example configuration of terminal 2a according to the embodiment. Note that terminals 2b and 2c have similar configurations and therefore their descriptions are omitted.
[0025] Terminal 2a comprises a control unit 20, which is composed of a CPU and the like, controls each part and executes various programs; a storage unit 21, which is composed of a storage medium such as flash memory and stores information; a communication unit 22, which communicates with the outside via the network 4; a 3D display 23; and an operation unit 24.
[0026] The control unit 20 functions as a data receiving means 200, a basic data reconstruction means 201, a rendering means 202, a display means 203, etc., by executing a streaming playback program 210, which will be described later as an information processing program.
[0027] The data receiving means 200 receives rendering data 112 (color data 113, depth data 114) from the server 1 via the communication unit 22 and stores them in the storage unit 21 as rendering data 211 (color data 212, depth data 213), respectively.
[0028] The basic data reconstruction means 201 reconstructs the basic data based on the rendering data 211 (color data 212, depth data 213) and stores it in the storage unit 21 as reconstructed basic data 214. The reconstructed basic data 214 is, for example, voxels or height maps. The type of data suitable for the reconstruction method and the reconstruction method will be described later.
[0029] The rendering means 202 processes the reconstruction base data 214, converts it from voxels and height maps to polygon meshes, and generates re-rendered data 215 based on a certain viewpoint (which may differ from the original viewpoint). In this embodiment, the re-rendered data 215 is in a format that matches the type of 3D display 23.
[0030] The display means 203 processes the re-rendered data 215 for display on the 3D display 23a.
[0031] The memory unit 21 stores a streaming playback program 210 that causes the control unit 20 to operate as each of the means 200203 described above, rendering data 211 (color data 212, depth data 213), reconstruction base data 214, re-rendering data 215, etc.
[0032] (Operation of the Information Processing Device) Next, the operation of this embodiment will be explained by dividing it into (1) basic operation, (2) streaming operation, and (3) reconstruction operation.
[0033] (1) Basic Operation As an example to explain the following operations, we will explain the case in which users 3a, 3b, 3c... operate the operation unit 24 of each terminal 2a, 2b, 2c... and execute a program such as a game provided by Server 1.
[0034] Server 1 executes programs such as games in which one or more participants play in a 3D space. Server 1 renders objects in its field of view from the viewpoints corresponding to terminals 2a, 2b, 2c, etc., and streams the rendered data.
[0035] Each terminal 2a, 2b, 2c... communicates with server 1 according to the operations of users 3a, 3b, 3c..., sending requests for changes to the movement of characters running on the program, as well as changes to the design of terrain, buildings, furniture, etc., and receiving rendering data streamed from server 1. In addition, each terminal 2a, 2b, 2c... reconstructs 3D data based on the streamed rendering data, renders the reconstructed 3D data, and outputs data compatible with the format of the 3D displays 23a, 23b, 23c.... The 3D displays 23a, 23b, 23c... display images that users 3a, 3b, 3c... can view in stereoscopic vision, and users 3a, 3b, 3c... view the images displayed on the 3D displays 23a, 23b, 23c....
[0036] Below, the operation in which Server 1 streams rendering data will be explained first in "(2) Streaming Operation," and the operation in which terminals 2a, 2b, 2c... reconstruct the basic data from the streaming data will be explained in "(3) Reconstruction Operation."
[0037] (2) Streaming Operation Diagram 4 is a flowchart for explaining an example of the operation of the information processing device.
[0038] First, the basic data processing means 100 of server 1 processes polygon mesh data as basic data 111 (S10), and generates rendering data 112A (color data 113A, depth data 114A) for each frame sequentially (S11).
[0039] Figures 5A to 5C are schematic diagrams illustrating the rendering and reconstruction of the underlying data.
[0040] Figure 5A is a diagram showing the relationship between the basic data 111 and the viewpoint v, where the depth data 114A is expressed as the distance from the viewpoint v to each point s00, s10, s20... of the target basic data 111. Here, in the first embodiment, the distance to the sides, back, bottom, etc. of the basic data 111 that are not visible from the viewpoint v is not included.
[0041] Figure 5B shows color data 113A and depth data 114A as part of rendering data 112A that represents the form of the basic data 111 as seen from viewpoint v. The color data 113A reflects the lighting effect (and effects such as camera distortion) on the material (texture) of the basic data 111. The color data 113A is displayed as an image (it is shown in monochrome due to the constraints of drawing representation, but it is generally displayed in color and may be monochrome), but it may of course be represented numerically. The depth data 114A does not contain information such as the coordinates of the viewpoint or the camera rotation angle, but it may include this information.
[0042] Next, the data distribution means 101 of server 1 packages the rendering data 112A (color data 113A, depth data 114A) of multiple frames and sequentially streams them (S12).
[0043] Next, the data receiving means 200 of terminal 2a sequentially receives rendering data 112A (color data 113A, depth data 114A) from server 1 via communication unit 22 (S20), and stores them in storage unit 21 as rendering data 211A (color data 212A, depth data 213A), respectively.
[0044] (3) Reconstruction Operation Next, the basic data reconstruction means 201 of the terminal 2a reconstructs the basic data for each frame based on the rendering data 211A (color data 212A, depth data 213A) (S21), and stores it in the storage unit 21 as the reconstructed basic data 214A as shown in FIG. 5C. Specifically, the reconstruction means 201 mainly predicts the shape of the object from the depth data 213A (the color data 212A may also be used), generates voxel data or height map data, and applies the color of the color data 212A to the data to generate the reconstructed basic data 214A. Since the lighting effect and the like are reflected in the material (texture) of the basic data 111 in the color data 113A, the basic data reconstruction means 201 may predict the material and lighting from the color data 212A using learning data or the like, and predict the correct material and color excluding the effects such as lighting and apply the color. Since the depth data 213A is expressed at a certain granularity, interpolation of surfaces and corners may be performed using filters, learning data, AI, or the like.
[0045] Also, when the depth data 114A also includes viewpoint information, the basic data reconstruction means 201 sets the viewpoint v in the 3D space, and sets a point at the position of the distance described in the depth data 213A in the direction of the line of sight determined from the rotation angle of the camera from the viewpoint v, and generates voxel data or height map data. Also, when the viewpoint v is fixed on the server 1 side, the terminal 2a side also shares this, and generates the reconstructed basic data 214A by the same process using the fixed viewpoint v.
[0046] Since the reconstructed basic data 214A reconstructed from the rendering data of one viewpoint is predicted to be planar, as an example, it is suitable to represent the reconstructed basic data 214A as a height map, and only the front and top surfaces of the object that can be reconstructed from the depth data 114A are included. That is, the side, back, and bottom surfaces of the basic data 111 are omitted.
[0047] Next, the rendering means 202 of the terminal 2a processes the reconstructed basic data 214A and generates the re-rendered data 215A based on a certain viewpoint (S22). In the case of this embodiment, since it is the re-rendered data 215A for display on the 3D display 23, for example, viewpoints for the right eye and the left eye are set, the reconstructed basic data 214A from the viewpoints is rendered to generate an image for the right eye and an image for the left eye, and the data is transmitted to the 3D display 23a as the re-rendered data 215A.
[0048] Next, the display means 203 of the terminal 2a performs display processing on the re-rendered data 215A on the 3D display 23a (S23), and the user 3a views the display result.
[0049] The basic data reconstruction means 201 generates the reconstructed basic data 214A for each frame of the streamed rendering data 211A. However, it may detect the difference for each frame of the streamed rendering data 211A and apply and update the difference to the reconstructed basic data 214A generated in the previous frame. Depending on the content of the streaming data (such as when the viewpoint does not change), it is also possible to reduce the processing burden of reconstructing the basic data by this update method.
[0050] (Operation of the First Embodiment) According to the above-described embodiment, since the depth data 114A is included in the rendering data 112A streamed and distributed by the server 1, the basic data can be reconstructed in the terminals 2a, 2b, 2c... based on the depth data 114A, and various processes using the reconstructed basic data 214A can be performed in the terminals 2a, 2b, 2c.... Even if the reconstructed basic data 214A does not include the side surface, the bottom surface, and the back surface in the case of the difference in viewpoints such as the degree of parallax between the left and right eyes used in the 3D displays 23a, 23b, 23c, it can function as the basic data without problems.
[0051] Furthermore, because terminals 2a, 2b, 2c, etc. are configured to reconstruct the basic 3D data, even if server 1 cannot output 3D format data that can be played back on 3D displays 23a, 23b, 23c, etc., the processing by terminals 2a, 2b, 2c, etc. will enable playback of the 3D data on the 3D displays 23a, 23b, 23c, etc.
[0052] [Second Embodiment] The second embodiment differs from the first embodiment in that it uses data from multiple viewpoints. The data can be from two or more viewpoints, but as an example, the case using data from three viewpoints will be described. Figures 6A to 6C are schematic diagrams illustrating the rendering and reconstruction of basic data according to the second embodiment. Note that the same configuration and operation as in the first embodiment will not be explained.
[0053] First, the basic data processing means 100 of server 1 processes polygon mesh data as basic data 111 (S10) and generates sequential rendering data 112B (color data 113B, depth data 114B) (S11). At this time, the rendering data 112B is generated based on multiple viewpoints. Multiple viewpoints may be set for one terminal, or rendering data obtained based on viewpoints of terminals other than terminal 2a, such as terminals 2b and 2c, may also be sent to terminal 2a. In the latter case, the cost of the generation process is not increased.
[0054] Figure 6A shows the relationship between the basic data 111 and viewpoints v1, v2, and v3. Depth data 1141, 1142, and 1143 are represented by distances l00, l10, l20, m00, m10, m20, n00, n10, n
[0055] Figure 6B shows color data 1131, 1132, 1133 and depth data 1141, 1142, 1143 as part of rendering data 112B that represents the basic data 111 form visible from viewpoints v1, v2, and v3.
[0056] Next, the data distribution means 101 of server 1 streams color data 1131, 1132, 1133 and depth data 1141, 1142, 1143 as rendering data 112B (S12).
[0057] Next, the data receiving means 200 of terminal 2a receives color data 1131, 1132, 1133 and depth data 1141, 1142, 1143 as rendering data 112B from server 1 via communication unit 22 (S20), and stores them in storage unit 21 as rendering data 211B (color data 2121, 2122, 2123, depth data 2131, 2132, 2133), respectively.
[0058] Next, the basic data reconstruction means 201 of terminal 2a reconstructs the basic data based on the rendering data 211B (color data 2121, 2122, 2123, depth data 2131, 2132, 2133) (S21), and stores it in the storage unit 21 as reconstructed basic data 214B, as shown in Figure 6C. Specifically, if each viewpoint v1, v2, and v3 are shared by terminal 2a, the basic data reconstruction means 201 can generate voxel data by projecting the depth data 2131, 2132, and 2133 from the direction of viewpoints v1, v2, and v3. Even if the viewpoints are not shared, the relative position of each viewpoint can be estimated from the information of each depth data, and voxel data can be generated by projecting the depth data 2131, 2132, and 2133 from the direction of the estimated viewpoint. Subsequently, the colors of the color data 2121, 2122, and 2123 are applied to the data to generate the reconstruction base data 214B. The reconstruction method is the same as in the first embodiment, but since depth data 2131, 2132, and 2133 from multiple directions are used, the shape can be reconstructed more accurately and can handle the base data 111 with complex shapes. Since the reconstruction is from multiple directions, voxel data is suitable for processing, but height maps may also be used.
[0059] (Effects of the second embodiment) According to the second embodiment described above, not only are the same effects as the first embodiment achieved, but because depth data 2131, 2132, and 2133 from multiple directions are used, the shape can be reconstructed more accurately, and it can also handle basic data 111 of complex shapes.
[0060] Furthermore, although the above-described case involves multiple viewpoints v1, v2, and v3 within a single frame, even in the case of only one viewpoint per frame, as in the first embodiment, if the viewpoint changes between multiple frames, the basic data may be reconstructed using the data from these viewpoints.
[0061] [Third Embodiment] The third embodiment differs from the first embodiment in that the depth data is extended in the depth direction. Figures 7A to 7C are schematic diagrams illustrating the rendering and reconstruction of the basic data according to the third embodiment. The same configuration and operation as in the first and second embodiments will not be explained.
[0062] First, the basic data processing means 100 of server 1 processes polygon mesh data as basic data 111 (S10) and generates sequential rendering data 112C (color data 113C, depth data 114C) (S11).
[0063] Here, depth data 114C is the distance to the surface of the base data 111 of the object visible from the viewpoint v, and to points inside the base data 111 that exist in the depth direction from the surface. However, it may also be the distance to the depth direction surfaces (sides, back, bottom, etc.) without including the internal points. Furthermore, color data 113C sets the color of each point on the surface and inside of the base data 111. The same color may be set for the inside as for the surface, or if no color is set for the inside, it may be estimated and set. Also, if the color data 113C reflects the effects of materials, lighting, etc., that color may be used as is for color data 113C, or the color without the effects may be estimated and used for color data 113C.
[0064] Figure 7A shows the relationship between the base data 111 and the viewpoint v, where the depth data 114C is represented by the distances l000, l100, l200... from the viewpoint v to each point s000, s100, s200... of the base data 111 of the object. It should be noted that points within the base data 111 not shown in Figure 7A also have distance information in the depth data 114C and color information in the color data 113C, and may also have information about objects located further back (objects that are hidden and not visible when actually rendered).
[0065] Figure 7B shows color data 113C and depth data 114C as part of rendering data 112C that represents the basic data 111 form visible from viewpoint v.
[0066] Next, the data distribution means 101 of server 1 streams color data 113C and depth data 114C as rendering data 112 (S12).
[0067] Next, the data receiving means 200 of terminal 2a receives rendering data 112 (color data 113C, depth data 114C) from server 1 via communication unit 22 (S20), and stores them in storage unit 21 as rendering data 211C (color data 212C, depth data 213C), respectively.
[0068] Next, the basic data reconstruction means 201 of terminal 2a reconstructs the basic data based on the rendering data 211C (color data 212C, depth data 213C) (S21), and stores it in the storage unit 21 as reconstructed basic data 214C as shown in Figure 7C. Specifically, the basic data reconstruction means 201 sets a viewpoint in 3D space, sets a point at a distance described in the depth data 213C from the viewpoint, generates voxel data, and applies the color of the color data 212C to the voxel data to generate the reconstructed basic data 214C. The cases in which the color data 212C is affected by materials, lighting, etc., and whether or not, are handled in the same way as in the first and second embodiments.
[0069] (Effects of the Third Embodiment) According to the third embodiment described above, since depth data 114C is acquired for all surfaces (and internal points) of the target, the basic data reconstruction means 201 can generate reconstructed basic data 214C more easily and accurately than in the first and second embodiments. Furthermore, if the reconstructed basic data 214C includes information on internal points, it is possible to handle cases where, as a result of performing an operation to change the shape of the reconstructed basic data 214C at terminals 2a, 2b, 2c..., it becomes necessary to display the inside of the reconstructed basic data 214C.
[0070] [Other Embodiments] The present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention.
[0071] For example, the above embodiment described an example of a 3D game running on server 1, but the invention can be applied to any system where 3D data is processed, rendered, and streamed on server 1, and then reconstructed and used on each terminal 2a, 2b, 2c, etc. For example, it can be applied to cases such as Augmented Reality (AR), where 3D data reconstructed from rendering data distributed from server 1 is superimposed on an image captured by a camera installed on the terminal for display processing. In particular, the present invention can be used when it is desired to distort the 3D data reconstructed from rendering data distributed from server 1. It can also be applied to projection mapping, where it is desired to project 3D data reconstructed from rendering data distributed from server 1 according to the shape of the projection target.
[0072] In the above embodiment, the functions of each means 100, 101, 200-203 of the control unit 10 were implemented by program, but all or part of each means may be implemented by hardware such as ASIC. Furthermore, the program used in the above embodiment can be stored and provided on a recording medium such as a CD-ROM. Also, the steps described in the above embodiment can be rearranged, deleted, or added without altering the essence of the present invention. Potential for industrial use
[0073] This invention provides an information processing program, terminal, information processing device, and information processing system that can reconstruct three-dimensional data on the receiving end, even when rendering and streaming three-dimensional data.
[0074] 1: Server 2a, 2b, 2c: Terminals 3a, 3b, 3c: Users 4: Network 10: Control Unit 11: Storage Unit 12: Communication Unit 20: Control Unit 21: Storage Unit 22: Communication Unit 23: 3D Display 24: Operation Unit 100: Basic Data Processing Means 101: Data Distribution Means 110: Streaming Program 111: Basic Data 112: Rendering Data 113: Color Data 114: Depth Data 200: Data Reception Means 201: Basic Data Reconstruction Means 202: Rendering Means 203: Display Means 210: Streaming Playback Program 211: Rendering Data 212: Color Data 213: Depth Data 214: Reconstructed Basic Data 215: Re-rendering Data
Claims
1. An information processing program for causing a computer to operate as a receiving means that receives color data representing the color of each pixel in an image and depth data representing the distance from the one viewpoint to the surface of the object, for one or more viewpoints, as rendering data generated by rendering basic data that represents an object as a three-dimensional object, which is streamed, based on one viewpoint; and a reconstruction means that reconstructs the basic data based on the received rendering data for one or more viewpoints.
2. The information processing program according to claim 1, further functioning as a rendering means for rendering the reconstructed basic data according to a playback device.
3. The information processing program according to claim 1, wherein the depth data expresses the distance of each point in the depth direction from the surface of the object based on the first viewpoint.
4. The reconstruction means is an information processing program according to claim 1, which estimates the target material from the color data.
5. An information processing program that causes a computer to function as rendering data generated by rendering basic data that represents an object as a three-dimensional object based on one viewpoint, including color data representing the color of each pixel in an image and depth data representing the distance from the viewpoint to the surface of the object, for one or more viewpoints, and as a distribution means for streaming the said rendering data.
6. A terminal having a receiving means for receiving color data representing the color of each pixel in an image and depth data representing the distance from the one viewpoint to the surface of the object, as rendering data generated by rendering basic data representing an object as a three-dimensional object, which is streamed, based on one viewpoint, for one or more viewpoints; and a reconstruction means for reconstructing the basic data based on the received rendering data for one or more viewpoints.
7. An information processing apparatus having processing means for generating rendering data, which is generated by rendering basic data for representing an object as a three-dimensional object based on one viewpoint, including color data representing the color of each pixel in the image and depth data representing the distance from the viewpoint to the surface of the object, for one or more viewpoints, and distribution means for streaming the said rendering data.
8. An information processing device having processing means for generating rendering data, which is generated by rendering basic data representing an object as a three-dimensional object based on one viewpoint, including color data representing the color of each pixel in the image and depth data representing the distance from the one viewpoint to the surface of the object, for one or more viewpoints; and distribution means for streaming the said rendering data; and a terminal having receiving means for receiving the streaming rendering data and reconstruction means for reconstructing basic data based on the received rendering data for one or more viewpoints.