Decoding device and method, and encoding method

Abstract

The present disclosure relates to a decoding device and method, and an encoding method, that make it possible to provide a wider variety of haptics experiences. The present disclosure executes a process including: decoding a bit stream to produce generation source haptics data, which is haptics data in a generation source; and producing receptor haptics data, which is haptics data sensed by a receptor, by using the generation source haptics data, a haptics propagation path length, which is the path length of a propagation path of the generation source haptics data from the generation source to the receptor, and a haptics propagation characteristic indicating the propagation characteristic of the haptics data achieved by a haptics propagation medium constituting the propagation path. The present disclosure can be applied to, e.g., an information processing device, an electronic apparatus, an encoding device, a decoding device, an information processing method, an encoding method, a decoding method, a program, or the like.

Description

Decoder device and method, and encoding method

[0001] The present disclosure relates to a decoder device and method, and an encoding method, and particularly to a decoder device and method and an encoding method adapted to provide more diverse haptic experiences.

[0002] Conventionally, there has been haptics technology for obtaining cutaneous feedback (so-called tactile sensation) by applying force, vibration, movement, etc. to a user (for example, see Patent Document 1). In such haptics technology, haptics data that a user (receiver) perceives with respect to a haptics device is provided, and the haptics device outputs the haptics data, thereby providing a haptic experience to the user.

[0003] WO 2019 / 013056

[0004] However, in the case of the method of supplying haptics data output by a haptics device to the haptics device as described above, there has been a possibility that the haptic experience that can be provided to a user is limited.

[0005] The present disclosure has been made in view of such a situation, and aims to provide more diverse haptic experiences.

[0006] A decoder device according to one aspect of the present technology includes a decoder that decodes a bit stream to generate source haptics data that is haptics data at a source, and the source haptics data, a haptics propagation path length that is the path length of the propagation path of the source haptics data from the source to a receiver, and a haptics propagation characteristic that indicates the propagation characteristic of haptics data by a haptics propagation medium that constitutes the propagation path, and a receiver haptics data generation unit that generates receiver haptics data that is haptics data perceived by the receiver.

[0007] One aspect of this technology is a decoding method that includes decoding a bitstream to generate source haptic data, which is haptic data at the source, and generating receptor haptic data, which is haptic data sensed by the receptor, using the source haptic data, the haptic propagation path length, which is the path length of the propagation path of the source haptic data from the source to the receptor, and the haptic propagation characteristics, which indicate the propagation characteristics of the haptic data by the haptic propagation medium constituting the propagation path.

[0008] Another aspect of this technology involves encoding a scene description, storing haptics-related information about haptics in the generated scene description, and encoding source haptics data, which is haptics data at the source.

[0009] In one aspect of this technology, the decoding device and method involve decoding a bitstream to generate source haptic data, which is haptic data at the source, and then using the source haptic data, the haptic propagation path length (the path length of the propagation path of the source haptic data from the source to the receptor), and the haptic propagation characteristics (which indicate the propagation characteristics of the haptic data by the haptic propagation medium constituting the propagation path) to generate receptor haptic data, which is haptic data sensed by the receptor.

[0010] In the encoding method for other aspects of this technology, a scene description is generated, haptics-related information about haptics is stored in the generated scene description, and source haptics data, which is haptics data at the source, is encoded.

[0011] This figure shows an example of how haptic data is provided. This figure shows an example of how haptic data is provided. This figure shows an example of how haptic data is played back. This figure shows an example of how haptic data is played back as it propagates through space. This figure shows an example of how haptic data is played back as it propagates through an object. This figure shows an example of a haptic propagation path through an object. This figure shows an example of how haptic propagation characteristics are controlled based on sensor values. This figure shows an example of how haptic propagation characteristics are controlled based on sensor values. This figure shows an example of haptic propagation characteristics for an avatar. This figure illustrates an example of how receptor haptic data is mixed. This figure illustrates an example of how haptic propagation characteristics are convolved. This figure shows an example of the main configuration of glTF2.0. This figure shows an example of glTF objects and reference relationships. This figure shows an example of how a scene description is written. This figure explains how to access binary data. This figure shows an example of how a scene description is written. This figure explains the relationship between buffer objects, buffer view objects, and accessor objects. This figure shows an example of how buffer objects, buffer view objects, and accessor objects are written. This figure explains an example of how objects are configured in a scene description. This figure shows an example of how a scene description is written. This is a diagram illustrating how to extend an object. This is a diagram illustrating the configuration of client processing. This is a diagram illustrating an example of an extension configuration for handling timed metadata. This is a diagram illustrating an example of a scene description. This is a diagram illustrating an example of a scene description. This is a diagram illustrating an example of an extension configuration for handling timed metadata. This is a diagram illustrating the main configuration of a client. This is a flowchart illustrating an example of the client processing flow. This is a diagram illustrating an example of how to provide haptics-related information. This is a diagram illustrating an example of haptics-related information. This is a diagram illustrating an example of haptics-related information. This is a diagram illustrating an example of haptics-related information. This is a block diagram illustrating the main configuration of a playback device. This is a flowchart illustrating an example of the playback processing flow.This is a flowchart illustrating an example of the playback process flow. This is a flowchart illustrating an example of the playback process flow. This is a flowchart illustrating an example of the playback process flow. This is a flowchart illustrating an example of the playback process flow. This is a flowchart illustrating an example of the playback process flow. This is a flowchart illustrating an example of the playback process flow. This is a flowchart illustrating an example of the playback process flow. This is a block diagram illustrating an example of the main configuration of a file generation device. This is a flowchart illustrating an example of the file generation process flow. This is a block diagram illustrating an example of the main configuration of a playback device. This is a flowchart illustrating an example of the playback process flow. This is a diagram illustrating an example of a method for playing back haptic data. This is a diagram illustrating an example of the configuration of a propagation path. This is a diagram illustrating an example of how receptor haptic data is derived. This is a diagram illustrating an example of the configuration of a propagation path. This is a diagram illustrating an example of how receptor haptic data is derived. This is a diagram illustrating an example of how receptor haptic data is derived. This is a diagram illustrating an example of how receptor haptic data is derived. This is a diagram illustrating an example of how receptor haptic data is derived. This is a diagram illustrating an example of propagation path configuration information. This is a diagram illustrating an example of propagation path configuration information. This is a diagram illustrating an example of the configuration of a propagation path. This is a diagram illustrating an example of how propagation path configuration information is described. This is a flowchart illustrating an example of the playback process flow. This figure shows an example of how haptic propagation media are used and controlled. This figure shows an example of how haptic propagation media are used and controlled. This figure shows an example of how haptic propagation media are used and controlled. This figure shows an example of propagation path control information. This figure shows an example of propagation path control information. This figure shows an example of propagation path control information. This figure shows an example of propagation path control information. This figure shows an example of propagation path control information. This figure shows an example of propagation path control information. This is a flowchart explaining an example of the regeneration process flow. This figure shows an example of receptor control information. This is a flowchart explaining an example of the regeneration process flow. This figure shows an example of linking receptors and haptic devices. This figure shows an example of linking receptors and haptic devices. This figure shows an example of receptor device linking information. This figure shows an example of how the linking between receptors and haptic devices is used. This figure shows an example of how the linking between receptors and haptic devices is used.This figure shows an example of the use of linking receptors and haptic devices. This figure shows an example of the use of linking receptors and haptic devices. This is a flowchart explaining an example of the playback process flow. This figure shows an example of haptic propagation characteristics. This figure shows an example of haptic propagation characteristics. This figure shows an example of haptic propagation characteristics. This is a block diagram showing an example of the main configuration of a computer.

[0012] The following describes the embodiments for implementing this disclosure. The description will be in the following order: 1. Supporting literature, etc., for technical content and technical terminology 2. Haptics technology 3. Updating haptic data using haptic propagation characteristics 4. Use of scene description 5. First embodiment (playback device) 6. Second embodiment (file generation device and playback device) 7. Application examples 8. Appendix

[0013] <1. Supporting Documents for Technical Content and Terminology> The scope disclosed in this technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent documents.

[0014] Patent document 1: (mentioned above) Non-patent document 1: Saurabh Bhatia, Patrick Cozzi, Alexey Knyazev, Tony Parisi, "Khronos glTF2.0", https: / / github.com / KhronosGroup / glTF / tree / master / specification / 2.0, June 9, 2017 Non-patent document 2: "Text of ISO / IEC CD 23090-14 Scene Description for MPEG Media", ISO / IEC JTC 1 / SC 29 / WG 3 N00485, 2021 / 10 / 12 Non-patent document 3: "Text of ISO / IEC FDIS 23090-14 2nd edition Scene description", ISO / IEC JTC 1 / SC 29 / WG 3 N01145, 2024-02-1, https: / / www.mpeg.org / wp-content / uploads / mpeg_meetings / 145_OnLine / w23512.zip Non-patent document 4: "Text for Committee Draft of ISO / IEC 23090-31: Haptics Coding", ISO / IEC JTC 1 / SC 29 / WG 7 N449, 2022-10-27,https: / / www.mpeg.org / wp-content / uploads / mpeg_meetings / 140_Mainz / w22111.zip

[0015] In other words, the content described in the aforementioned patent and non-patent documents, as well as the content of other documents referenced in those documents, will also serve as a basis for determining the support requirements.

[0016] <2. Haptics Technology> Conventionally, as described in Patent Document 1, for example, there has been haptics technology that provides tactile feedback (so-called touch) by applying force, vibration, movement, etc., to the user. In such haptics technology, haptic data perceived by the user (receptor) is provided to a haptics device, and the haptics device outputs that haptic data, thereby providing the user with a haptic experience.

[0017] In this specification, "haptic experience" refers to a tactile experience obtained through haptic technology. A "haptic device" is a device that converts haptic data into physical motion using actuators, etc., and outputs physical motion corresponding to the haptic data. "Outputting physical motion" means, for example, generating vibrations, etc., in response to haptic data. Therefore, it can also be said to represent physical motion (physical motion representation). The output of a haptic device may also be called "haptic output" or "physical motion output." Furthermore, an entity that senses this output of a haptic device is also called a "haptic listener." For example, a user wearing a haptic device (or the part of the user's body to which the haptic device is attached) can be considered a listener.

[0018] Furthermore, "haptic data" refers to data that provides a haptic experience. For example, vibration signals may be included in haptic data. As mentioned above, haptic data is used to drive haptic devices, so it can also be said to be a control signal for the haptic device (or its actuator). Media that uses haptic data is also called haptic media. In other words, "haptic media" refers to media that provides a haptic experience using haptic data. Content that uses haptic data is also called haptic content. In other words, "haptic content" refers to content that provides a haptic experience using haptic data.

[0019] Haptics technology simulates how receptors detect haptic data generated by a virtual haptic event in a given 3D area (which can be 1D or 2D). Here, a "haptic event" refers to the generation of haptic data. Therefore, this "haptic event" includes not only interactive events such as collisions, but also events synchronized with some information, such as time. In this specification, the location where this haptic event occurs is also referred to as the "Haptic Source." For example, if the haptic data is a vibration signal, this source can also be called a vibration source.

[0020] In other words, in this simulation, haptic data generated by a haptic event propagates from the source to the receptor. In this specification, the location of the source is also referred to as the "source location." The location of the receptor is also referred to as the "receptor location." That is, the haptic device represents the physical motion corresponding to the haptic data at the receptor location. In this specification, the propagation path from the source location to the receptor location through which the haptic data propagates is also referred to as the "haptic propagation path." The medium constituting this haptic propagation path is also referred to as the "haptic propagation medium." The length of this haptic propagation path is also referred to as the "haptic propagation path length."

[0021] Generally, this haptic data (the physical motion corresponding to the haptic data) changes as it propagates. For example, in the case of vibration, the amplitude decreases as you move away from the source. That is, the haptic propagation path (haptic propagation medium) has predetermined propagation characteristics, and the propagated haptic data changes according to those propagation characteristics. Such characteristics related to the propagation of haptics are also called "haptic propagation characteristics." Generally, these haptic propagation characteristics are attenuation characteristics.

[0022] In other words, the haptic data (physical motion corresponding to the haptic data) sensed by a receptor can change depending on factors such as the positional relationship between the source and the receptor. However, conventional methods of supplying haptic data output by haptic devices could not change the haptic data in response to changes in that positional relationship. This meant that the haptic experience that could be provided to the user was limited.

[0023] For example, as shown in Figure 1, suppose we simulate how a user 12 receives a virtual shock wave generated at a source 11 using haptics technology. User 12 wears haptic devices at multiple locations such as the head, torso, hands, and feet, and the shock wave (vibration) received at each part is represented by each haptic device. In this case, the magnitude of the shock wave received by each part may change depending on the position of each part. Therefore, in the conventional method, it is necessary to supply haptic data to each haptic device according to its position (receptor position), such as haptic data 13, haptic data 14, and haptic data 15.

[0024] However, in this case, there are infinitely many possible patterns for receptor positions (the positions of haptic devices). Therefore, it was difficult to prepare and supply optimal haptic data for all receptor positions.

[0025] Furthermore, for example, as shown in Figure 2, let's say we use haptic technology to simulate how the user's hand 23 receives an impact (vibration) caused by a collision between a virtual sphere 21 and a virtual rod 22. In this case, the source is the point where the sphere 21 and the rod 22 collide. However, it is not predetermined where the sphere 21 collides with the rod 22. For example, the sphere 21 could collide with point 24 on the rod 22, or with point 25, or with point 26. In other words, any of points 24 through 26 could be the source. Since the distance from each point to the hand 23 is different, the magnitude of the impact (vibration) received by the hand 23 can change depending on which of these points is the source. Therefore, in the conventional method, for example, haptic data 27 for when point 24 is the source, haptic data 28 for when point 25 is the source, and haptic data 29 for when point 26 is the source are prepared, and the haptic data to be provided to the haptic device must be selected depending on where the sphere 21 collides.

[0026] However, in this case, there are infinitely many possible collision locations (i.e., source locations) other than points 24 to 26 mentioned above. Therefore, it was difficult to prepare and supply optimal haptic data for all source locations.

[0027] In other words, with conventional methods, it was practically only possible to provide haptic data corresponding to a limited number of spatial patterns. Therefore, the haptic experiences that could be provided to users were limited, potentially reducing the realism of the haptic experience (i.e., making it less accurate). In short, the quality of the haptic experience was at risk. Furthermore, attempting to provide a wider variety of haptic experiences would require preparing and providing more haptic data, potentially increasing processing load and costs.

[0028] <3. Updating Haptics Data Using Haptics Propagation Characteristics> <Method 1> As shown in the top row of the table in Figure 3, receptor haptics data is generated using source haptics data, haptics propagation path length, and haptics propagation characteristics (Method 1). Here, "source haptics data" refers to the haptics data generated at the source. Also, "receptor haptics data" refers to the haptics data sensed by the receptor. In other words, receptor haptics data can also be said to be the haptics data input to the haptics device.

[0029] For example, a decoding device includes a decoding unit that decodes a bitstream to generate source haptic data, which is haptic data at the source, and a receptor haptic data generation unit that uses the source haptic data, the haptic propagation path length, which is the path length of the propagation path of the source haptic data from the source to the receptor, and the haptic propagation characteristics, which indicate the propagation characteristics of the haptic data by the haptic propagation medium constituting the propagation path, to generate receptor haptic data, which is haptic data sensed by the receptor.

[0030] For example, a decoding method performed by a decoding device includes decoding a bitstream to generate source haptic data, which is haptic data at the source, and generating receptor haptic data, which is haptic data sensed by the receptor, using the source haptic data, the haptic propagation path length, which is the path length of the propagation path of the source haptic data from the source to the receptor, and the haptic propagation characteristics, which indicate the propagation characteristics of the haptic data by the haptic propagation medium constituting the propagation path.

[0031] For example, a program may cause a computer to perform a process that includes decoding a bitstream to generate source haptics data, which is haptic data at the source, and then using that source haptics data, the haptic propagation path length (the path length of the propagation path of the source haptics data from the source to the receptor), and the haptic propagation characteristics (which indicate the propagation characteristics of the haptic data by the haptic propagation medium constituting the propagation path) to generate receptor haptics data, which is haptic data sensed by the receptor.

[0032] By using haptic propagation path length and haptic propagation characteristics in this way, it is possible to generate more diverse receptor haptic data from source haptic data. In other words, it is possible to provide a more diverse haptic experience. Furthermore, the amount of transmitted data can be reduced. In addition, it is possible to present highly immersive haptics with low latency. Moreover, if data is sent and buffered in advance, the amount of memory required can also be reduced. Furthermore, it becomes unnecessary to prepare separate data for each multi-point haptic presentation.

[0033] The decoding device may further include a haptic propagation path length derivation unit that derives the haptic propagation path length. The receptor haptic data generation unit may then generate receptor haptic data using the derived haptic propagation path length. The decoding device may further include a receptor haptic data output unit that outputs the generated receptor haptic data.

[0034] <Method 1-1> When Method 1 is applied, space may be used as the haptic propagation medium, as shown in the second row from the top of the table in Figure 3. For example, in a decoding device, space may be the haptic propagation medium. The receptor haptic data generation unit may then generate receptor haptic data using source haptic data, an arbitrary haptic propagation path length within that space, and haptic propagation characteristics based on the components or state of that space.

[0035] For example, as shown in Figure 4, suppose haptic data generated at source location 101 is perceived by user 102 at receptor locations 103-0, 103-1, and 103-2. In this case, using a haptic propagation characteristic F(r) that depends on the haptic propagation path length r, receptor haptic data 105-0 for receptor location 103-0 is generated from source haptic data 104, receptor haptic data 105-1 for receptor location 103-1 is generated, and receptor haptic data 105-2 for receptor location 103-2 is generated. In this way, receptor haptic data for any receptor location can be generated and provided. Therefore, the haptic experience that can be provided to the user is not limited, and a wider variety of haptic experiences can be provided. That is, a reduction in the reality of the haptic experience can be suppressed (i.e., a reduction in the accuracy of the haptic experience can be suppressed), and a reduction in the quality of the haptic experience can be suppressed. In other words, it is possible to suppress increases in processing load and costs.

[0036] In this case, the haptic propagation medium is composed of a space (non-object) made up of, for example, a gas or liquid. Furthermore, the haptic propagation characteristics depend on the components and state that constitute the space (the components that constitute the space, or the state of the space, or both; and may also include other elements). These haptic propagation characteristics can be anything. For example, they may be expressed using a predetermined function, or they may be expressed without using a function. For example, as shown in equation (1) below, they may be designed to not attenuate (the source haptic data is transmitted as is) (noAttenuation).

[0037] ... (1)

[0038] Furthermore, as a haptic propagation characteristic, a function that attenuates inversely proportional to the distance from the source (inverseDistance), as shown in equation (2) below, may be applied.

[0039] ... (2)

[0040] Also, as the haptics propagation characteristic, a function (LinearDistance) that decays in proportion to the distance from the source position, such as the following equation (3), may be applied.

[0041] ・・・(3)

[0042] Also, as the haptics propagation characteristic, a function (exponentialDistance) that decays exponentially with respect to the distance from the source position, such as the following equation (4), may be applied.

[0043] ・・・(4)

[0044] Of course, functions other than these may be applied. It may be possible to arbitrarily set the function. In the above equations (1) to (4), md indicates "maximum distance", rf indicates "roll factor", rd indicates "reference distance (distance for gain 1 setting)", and d indicates "distance".

[0045] Note that when the haptics propagation medium is space in this way, the haptics propagation path length may indicate the length of the propagation path in a 3D region (1D or 2D may also be used). For example, the haptics propagation path length may be the straight-line distance between the source position and the receiver position, or it may be something else. For example, it may take into account reflection, wrapping, etc. For example, the haptics propagation path is calculated based on the acquired or specified source position and receiver position.

[0046] <Method 1-2> When Method 1 is applied, as shown in the third row from the top of the table in FIG. 3, the object may be used as a haptics propagation medium (Method 1-2). For example, in the decoding device, the haptics propagation medium may be an object. Then, the receiver haptics data generation unit may generate receiver haptics data using the source haptics data, the haptics propagation path length corresponding to the shape of the object, and the haptics propagation characteristics depending on the material, shape, or state of the object. By doing so, it is possible to represent haptics propagation not only through space but also through an object such as the ground.

[0047] For example, as shown in FIG. 5, assume that the impact (vibration) received by the user's hand 113 due to the virtual sphere 111 colliding with the virtual rod 112 is simulated using haptics technology. In this case, using the haptics propagation characteristics F(r) depending on the haptics propagation path length r, from the source haptics data 115, the receiver haptics data 116-0 when the sphere 111 and the rod 112 collide at the source position 114-1 is generated, the receiver haptics data 116-1 when the sphere 111 and the rod 112 collide at the source position 114-2 is generated, and the receiver haptics data 116-2 when the sphere 111 and the rod 112 collide at the source position 114-3 is generated. In this way, it is possible to generate and provide the receiver haptics data corresponding to any source position. Therefore, the haptics experience that can be provided to the user is not limited, and a more diverse haptics experience can be provided. That is, it is possible to suppress the reduction in the reality of the haptics experience (that is, suppress the reduction in the accuracy of the haptics experience) and suppress the reduction in the quality of the haptics experience. In other words, it is possible to suppress an increase in processing load and cost.

[0048] In this case, the haptic propagation medium is composed of an object made of, for example, a solid or a liquid. The haptic propagation characteristics depend on the material, shape, state, etc., of the object (including at least one of the material, shape, or state of the space surrounding it; other elements may also be included). These haptic propagation characteristics can be anything. For example, they may be expressed using a predetermined function, as in equations (1) to (4) above, or they may be expressed without a function. Other functions may also be applied. The function may be set arbitrarily.

[0049] When the haptic propagation medium is an object, the haptic propagation path is either inside or on the surface of the object. Therefore, the haptic propagation path (haptic propagation path length) depends on the shape of the object. The haptic propagation path may be 1D information, 2D information, or 3D information. For example, as shown in Figure 6, if the shape of object 121 is a function of V=u^2, the length of the haptic propagation path 124 between the source position 122 and the receptor position 123 can be expressed as r=sqrt(u^2+V^2).

[0050] <Method 1-3> When Method 1 is applied, the haptic propagation characteristics may be set as shown in the fourth row from the top of the table in Figure 3 (Method 1-3). For example, the decoding device may further include a haptic propagation characteristics setting unit for setting the haptic propagation characteristics. The receptor haptic data generation unit may then generate receptor haptic data using the set haptic propagation characteristics.

[0051] By setting haptic propagation characteristics in this way, it is possible to more accurately represent the propagation of diverse haptic data than when using predetermined haptic propagation characteristics. In other words, differences in materials can be represented by setting haptic propagation characteristics.

[0052] The method for setting these haptic propagation characteristics can be any method. For example, a list of candidate functions may be prepared, and a function to be applied may be selected from among these candidates. For example, in a decoder, the haptic propagation characteristic setting unit may select a function to be used as the haptic propagation characteristic from among several candidates. For example, haptic propagation characteristics such as those shown in equations (1) to (4) above may be prepared as candidates, and a haptic propagation characteristic to be applied may be selected from among them. Furthermore, variables may be specified.

[0053] Alternatively, haptic propagation characteristics may be set by setting coefficients (for each frequency) of a predetermined model, such as RT60 or an FIR (Finite Impulse Response) filter. For example, in a decoding device, a haptic propagation characteristic setting unit may set the coefficients of a predetermined model of haptic propagation characteristics. The reverberation time of RT60 is an acoustic parameter of the main room and, according to ISO (International Organization for Standardization) 3382, indicates the time required for the acoustic energy of the room to decrease by 60 dB after source emissions have stopped. The value of RT60 ranges from a few seconds to several seconds and varies depending on the size of the room and the properties of the materials used in its construction.

[0054] <Method 1-4> When Method 1 is applied, the haptic propagation characteristics may be set using a sensor, as shown in the fifth row from the top of the table in Figure 3 (Method 1-4). For example, the decoding device may further include a detection unit that detects predetermined information about the external environment. The decoding device may further include a haptic propagation characteristic setting unit that sets the haptic propagation characteristics based on the detected value.

[0055] For example, a sensor may be used to detect the pressing force F, contact area S, pressure P, etc., and the haptic propagation characteristics may be set based on these detected values. By doing so, it is possible to set haptic propagation characteristics that are more suitable for the situation, thereby suppressing a reduction in the realism of the haptic experience (i.e., suppressing a reduction in the accuracy of the haptic experience) and suppressing a reduction in the quality of the haptic experience.

[0056] This process may be performed by a regeneration device (PE) or by a haptic device. By having the regeneration device (PE) perform this process, an increase in the processing load on the haptic device can be suppressed. Conversely, by having the haptic device perform this process, an increase in the processing load on the regeneration device (PE) can also be suppressed.

[0057] <Method 1-4-1> When Method 1-4 is applied, the contact state between the object and the user in the scene may be detected, as shown in the sixth row from the top of the table in Figure 3, and the haptic propagation characteristics may be set using the detection results (Method 1-4-1).

[0058] In this specification, "scene" refers to a 3D space in which information from a virtual world is simulated. This 3D space may be a virtual space or an augmented reality space in which information from a virtual world is superimposed. In other words, an "object" in a "scene" may be an object of virtual information (virtual object) or an object that exists in the real world (real object). Similarly, a "user" in a "scene" may be virtual information (avatar) or the user's own body in the real world. In other words, the "state of contact between an object and a user in a scene" may be, for example, the state of contact between a virtual object and an avatar, the state of contact between the user's body and a virtual object, or the state of contact between the user's body and a real object.

[0059] However, this "contact state between the object and the user in the scene" merely represents how the object and the user interact in the simulation of information in the virtual world. Therefore, the detection result of this contact state does not necessarily have to be the detection signal obtained by a sensor device in the real world. For example, information from the real world (e.g., the user's gripping force) may be detected by a sensor device, and based on that information, the contact state in the simulation (e.g., the amount of force the avatar uses to grip the virtual ball) may be derived as the "detection result." Alternatively, this "detection result" may be derived solely through simulation (calculation) in the virtual world, without any detection by a sensor device in the real world.

[0060] For example, in a decoding device, a detection unit may detect information regarding the contact state between an object and a user in a scene. Then, a haptics propagation characteristic setting unit may set the haptics propagation characteristics based on the contact state indicated by the detected value. For example, pressure P or contact area may be detected as part of the contact state, and the haptics propagation characteristics may be set according to the detected value.

[0061] For example, the contact state can be determined by the pressure p when the device is gripped from the sensor, and a custom vibration propagation characteristic function including pressure p can be created and used to generate a signal that propagates strongly when the device is gripped tightly. For example, as shown in Figure 7A, if the contact area between the object 131 and the user's avatar 132 is small in the scene, the attenuation rate of the haptic data propagated from the source position 133 to the receptor position 134 may be increased. Also, as shown in Figure 7B, if the contact area between the object 131 and the user's avatar 132 is large in the scene, the attenuation rate of the haptic data propagated from the source position 133 to the receptor position 134 may be decreased. In other words, as shown in Figure 7C, in addition to an amplifier that shows pre-gain and an amplifier that shows attenuation, an amplifier that shows propagation characteristics that reflect the sensor value may be applied as the haptic propagation characteristic. By doing so, the sense of presence can be improved.

[0062] <Method 1-4-2> When Method 1-4 is applied, the collision behavior of objects in the scene may be detected and the haptic propagation characteristics may be set using the detection results, as shown in the seventh row from the top of the table in Figure 3 (Method 1-4-2).

[0063] This "description of object collisions in a scene" is similar to the "state of contact between an object and the user in a scene" described above, and it shows how objects collide in a simulation of information in the virtual world. In other words, in this case as well, the "objects" in the "scene" may be objects of virtual information (virtual objects) or objects that exist in real space (real objects). Similarly, the "collisions" in the "scene" may be virtual events or events that actually occur in real space. Furthermore, the detection result of these collisions does not need to be the detection signal from a sensor device in real space itself.

[0064] For example, in a decoding device, the detection unit may detect information regarding the collision of objects in the scene. Alternatively, the haptic propagation characteristic setting unit may set the haptic propagation characteristics based on the collision characteristics indicated by the detected values. For example, a custom vibration propagation characteristic function may be created by correlating the relative velocity value with the object immediately before contact with the UI sensor value, and this can be used to set the vibration intensity during collision. In this way, haptic data for each velocity becomes unnecessary.

[0065] <Method 1-4-3> When Method 1-4 is applied, the contact state between the haptics device and the user in real space may be detected, as shown in the eighth row from the top of the table in Figure 3, and the haptics propagation characteristics may be set using the detection results (Method 1-4-3). For example, in the decoding device, the detection unit may detect information regarding the contact state between the haptics device and the user in real space. The haptics propagation characteristics setting unit may then set the haptics propagation characteristics based on the contact state indicated by the detected value. For example, pressure P, contact area, correctness of installation, etc., may be detected as the contact state, and the haptics propagation characteristics may be set according to the detected value. For example, the device contact state in the real world may be detected using sensor values, and a custom haptics propagation characteristic function that corrects (cancels defects) so that the original haptics data is transmitted appropriately may be set and used for the conversion of haptics data. By doing so, the sense of presence can be improved and the quality of the haptics experience can be stabilized.

[0066] For example, as shown in Figure 8A, if user 141 has attached the haptic device 142 in the correct position and user 141 and haptic device 142 are in firm contact, the haptic propagation characteristics may be set to reduce the amount of haptic data. Conversely, as shown in Figure 8B, if user 141 has not attached the haptic device 142 in the correct position and user 141 and haptic device 142 are not in firm contact, the haptic propagation characteristics may be set to increase the amount of haptic data. In other words, as shown in Figure 8C, in addition to amplifiers that show pre-gain and amplifiers that show attenuation, an amplifier that shows propagation characteristics that reflect sensor values ​​may be applied as the haptic propagation characteristics. By doing so, the sense of presence can be improved.

[0067] <Method 1-5> When Method 1 is applied, receptor haptics data may be generated using a sensor, as shown in the ninth row from the top of the table in Figure 3 (Method 1-5). In other words, the external environment may be detected in the same way as in Method 1-4 described above, and the receptor haptics data may be generated or updated using the detection results. For example, the decoding device may further include a detection unit that detects predetermined information about the external environment. Then, the receptor haptics data generation unit may generate receptor haptics data based on the detected values.

[0068] For example, sensors may be used to detect pressing force F, contact area S, pressure P, etc., and receptor haptics data may be generated based on these detected values. By doing so, receptor haptics data more appropriate to the situation can be generated, thereby suppressing a reduction in the realism of the haptic experience (i.e., a reduction in the accuracy of the haptic experience) and suppressing a reduction in the quality of the haptic experience.

[0069] This process may be performed by a regeneration device (PE) or by a haptic device. By having the regeneration device (PE) perform this process, an increase in the processing load on the haptic device can be suppressed. Conversely, by having the haptic device perform this process, an increase in the processing load on the regeneration device (PE) can also be suppressed.

[0070] <Method 1-5-1> When Method 1-5 is applied, the contact state between an object and the user in the scene may be detected, as shown in the 10th row from the top of the table in Figure 3, and receptor haptics data may be generated using the detection results (Method 1-5-1). The "contact state between an object and the user in the scene" is the same as in the case of Method 1-4-1 described above. For example, in the decoding device, the detection unit may detect information regarding the contact state between an object and the user in the scene. The receptor haptics data generation unit may then generate receptor haptics data based on the contact state indicated by the detected value. For example, pressure P, contact area, etc. may be detected as the contact state, and receptor haptics data may be generated according to the detected value.

[0071] For example, the contact state can be determined by the pressure p when the device is grasped from the sensor, and receptor haptic data may be generated to reflect the pressure p that generates a strongly propagating signal when the device is grasped tightly. For example, as shown in Figure 7A, if the contact area between the object 131 and the user's avatar 132 is small in the scene, the attenuation rate of the receptor haptic data propagated from the source position 133 to the receptor position 134 may be increased. Also, as shown in Figure 7B, if the contact area between the object 131 and the user's avatar 132 is large in the scene, the attenuation rate of the receptor haptic data propagated from the source position 133 to the receptor position 134 may be decreased. By doing so, the sense of presence can be improved.

[0072] <Method 1-5-2> When Method 1-5 is applied, the collision of objects in the scene may be detected and receptor haptics data may be generated using the detection results, as shown in the 11th row from the top of the table in Figure 3 (Method 1-5-2). The "collision of objects in the scene" is the same as in the case of Method 1-4-2 described above. For example, in the decoding device, the detection unit may detect information regarding the collision of objects in the scene. Alternatively, the receptor haptics data generation unit may generate receptor haptics data based on the collision characteristics indicated by the detected values. For example, receptor haptics data may be generated by correlating the relative velocity value with the object immediately before contact with the UI sensor value and used to set the vibration intensity at the time of collision. In this way, velocity-specific haptics data is not required.

[0073] <Method 1-5-3> When Method 1-5 is applied, as shown in the 12th row from the top of the table in Figure 3, the contact state between the haptic device and the user in real space may be detected, and receptor haptic data may be generated using the detection results (Method 1-5-3). For example, in the decoding device, the detection unit may detect information regarding the contact state between the haptic device and the user in real space. The receptor haptic data generation unit may then generate receptor haptic data based on the contact state indicated by the detected value. For example, pressure P, contact area, correctness of installation, etc., may be detected as the contact state, and receptor haptic data may be generated according to the detected value. For example, the device contact state in the real world may be detected using sensor values, and the receptor haptic data may be corrected (defect cancellation) so that the original haptic data is transmitted appropriately. By doing so, the sense of presence can be improved, and the quality of the haptic experience can be stabilized.

[0074] For example, as shown in Figure 8A, if user 141 is wearing the haptic device 142 in the correct position and user 141 and haptic device 142 are in firm contact, the receptor haptic data may be made smaller. Conversely, as shown in Figure 8B, if user 141 is not wearing the haptic device 142 in the correct position and user 141 and haptic device 142 are not in firm contact, the receptor haptic data may be made larger. By doing so, the sense of presence can be improved.

[0075] <Method 1-6> When Method 1 is applied, haptic propagation characteristics may be set for the avatar as shown in the 13th row from the top of the table in Figure 3 (Method 1-6). For example, in the decoding device, the receptor haptic data generation unit may generate receptor haptic data using haptic propagation characteristics that indicate the propagation characteristics of haptic data by the avatar. By doing so, it is possible to reflect differences in haptic propagation characteristics and occlusion caused by the user's body, and to allow the user to experience different bodily presences in the virtual space.

[0076] For example, as shown in Figure 9A, haptic propagation characteristics may be set for the avatar 151 that acts as a substitute for the user in the virtual space (scene). Note that these haptic propagation characteristics may differ for each part of the avatar 151.

[0077] In this case, the haptic propagation characteristics represent the attenuation caused by propagation between the source location and the receptor location via the avatar or the user themselves. These haptic propagation characteristics may be for creating a sense of reality or for creating a virtual experience. For example, the overall haptic characteristics of the avatar 151 may be set, or the haptic characteristics of each part of the avatar 151 may be set. This is particularly effective in cases where one user wears multiple haptic devices.

[0078] For example, the haptic propagation characteristics of this avatar may represent how haptic data propagates through the avatar and the user's body. For instance, as shown in Figure 9B, when representing haptic data generated at the source position 153 of the avatar 152 in a haptic device at the receptor position 154, applying the haptic propagation characteristics set for the avatar 152 can make the representation more accurate. This can be applied, for example, to a case where vibrations received by the right hand are detected by a haptic device worn on the left hand.

[0079] Furthermore, as shown in Figure 9C, when haptic data generated at the source position 153 of the avatar 152 is represented by a haptic device attached to the receptor position 154-1 and a haptic device attached to the receptor position 154-2, applying the haptic propagation characteristics set for the avatar 152 allows for a more accurate representation of the differences in physical motion (haptic data supplied to each haptic device) represented by each haptic device. For example, this can be applied to cases where vibrations received by the right hand are detected by haptic devices attached to the torso or left hand.

[0080] Furthermore, as shown in the example in Figure 9C, it is possible to prevent haptic data generated at the source position 153 of the avatar 152 from propagating to the receptor position 154. In other words, the blocking characteristics (directivity of haptic data) by the avatar or the user's body may be represented. For example, an impact received at the stomach may not be transmitted to a haptic device attached to the back.

[0081] <Method 1-7> When Method 1 is applied, mixed receptor haptic data, which have been propagated along different haptic propagation paths, may be mixed to generate mixed receptor haptic data, as shown in the 14th row from the top of the table in Figure 3 (Method 1-7). For example, in a decoding device, a receptor haptic data generation unit may generate multiple receptor haptic data, which have been propagated along different haptic propagation paths, and then mix the generated multiple receptor haptic data to generate mixed receptor haptic data.

[0082] For example, as shown in Figure 9, multiple receptor haptic data may be mixed in a mixer. This method may be applied, for example, to cases where haptic data propagates from one source through different haptic propagation media. Alternatively, this method may be applied to cases where multiple haptic data propagate from different sources. The data may also be mixed for each frequency component. Furthermore, the mixing ratio (MixRatio) for each data may be set. By doing so, the more complex propagation of haptic data can be represented more accurately.

[0083] <Method 1-8> When Method 1 is applied, multiple haptic propagation characteristics may be combined as shown in the 15th row from the top of the table in Figure 3 (Method 1-8). For example, the decoding device may further include a haptic propagation characteristic convolution unit that derives a convolutional haptic propagation characteristic by convolving multiple haptic propagation characteristics. Then, the receptor haptic data generation unit may generate receptor haptic data using the derived convolutional haptic propagation characteristic.

[0084] By doing so, for example, the propagation characteristics when haptic data propagates through multiple haptic propagation media can be represented more accurately. In other words, by superimposing several types of haptic data, a more realistic haptic experience can be provided. Furthermore, the effort required to acquire superimposed data is eliminated.

[0085] <Method 1-8-1> When Method 1-8 is applied, the convolutional haptics propagation characteristics may be retained and used, as shown in the 16th row from the top of the table in Figure 3 (Method 1-8-1). For example, in a decoding device, the haptics propagation characteristic convolution unit may retain the derived convolutional haptics propagation characteristics. The receptor haptics data generation unit may then generate receptor haptics data using the retained convolutional haptics propagation characteristics. In other words, as shown in Figure 11A, the convolution calculation result may be retained as a single haptics propagation characteristic and used to generate receptor haptics data. By doing so, the calculation using the haptics propagation characteristics can be performed only once, and the increase in computational load can be suppressed. Furthermore, processing can be performed at a faster speed.

[0086] <Method 1-8-2> When Method 1-8 is applied, as shown in the bottom row of the table in Figure 3, each haptic propagation characteristic may be retained and the convolutional haptic propagation characteristic may be derived when in use (Method 1-8-2). For example, in a decoding device, the haptic propagation characteristic convolution unit may retain multiple haptic propagation characteristics and derive the convolutional haptic propagation characteristic by convolving the retained multiple haptic propagation characteristics according to the propagation path. Then, the receptor haptic data generation unit may generate receptor haptic data using the derived convolutional haptic propagation characteristic. In other words, as shown in Figure 11B, when using the convolution result, the convolution operation may be performed each time according to the situation to derive the convolutional haptic propagation characteristic. By doing so, calculations using haptic propagation characteristics can be performed in another location, making it possible to share the load. In addition, it becomes easier to make changes when the haptic propagation path changes.

[0087] <4. Use of Scene Descriptions> <gltf2.0> Conventionally, as described in Non-Patent Document 1, for example, there was glTF (The GL Transmission Format) (registered trademark) 2.0, a format for arranging 3D (three-dimensional) objects in three-dimensional space. glTF2.0 consists of a JSON format file (.glTF), a binary file (.bin), and an image file (.png, .jpg, etc.), as shown in Figure 12, for example. The binary file stores binary data such as geometry and animation. The image file stores data such as textures.

[0088] A JSON format file is a scene description file written in JSON (JavaScript® Object Notation). A scene description is metadata that describes a scene in 3D content. The description of this scene defines what kind of scene it is. A scene description file is a file that stores such a scene description. In this disclosure, a scene description file is also referred to as a scene description file.

[0089] A JSON format file is composed of a list of key-value pairs. An example of this format is shown below: "KEY":"VALUE"

[0090] The key consists of a string. The value consists of a number, string, boolean, array, object, or null, etc.

[0091] Furthermore, multiple key-value pairs ("KEY":"VALUE") can be grouped together using curly braces {}. This grouped structure is also called a JSON object. An example of its format is shown below: "user":{"id":1, "name":"tanaka"}

[0092] In this example, a JSON object is defined that combines the pairs "id":1 and "name":"tanaka" as the values ​​corresponding to the key (user).

[0093] Additionally, zero or more values ​​can be enclosed in square brackets [] to create an array. This array is also called a JSON array. For example, a JSON object can be used as an element of this JSON array. An example of its format is shown below: `test":["hoge", "fuga", "bar"]` `users":[{"id":1, "name":"tanaka"},{"id":2,"name":"yamada"},{"id":3, "name":"sato"}]`

[0094] Figure 13 shows the glTF objects that can be placed at the top level of a JSON format file and the reference relationships they can have. In the tree structure shown in Figure 13, the elongated ovals represent objects, and the arrows between those objects indicate the reference relationships. As shown in Figure 13, objects such as "scene", "node", "mesh", "camera", "skin", "material", and "texture" are placed at the top level of a JSON format file.

[0095] An example of such a JSON format file (scene description) is shown in Figure 14. The JSON format file 220 in Figure 14 shows an example of a portion of the top level description. In this JSON format file 220, all top-level objects 221 used are described at the very top. These top-level objects 221 are the glTF objects shown in Figure 13. In addition, in the JSON format file 20, reference relationships between objects are shown as indicated by the arrow 222. More specifically, these reference relationships are shown by specifying the index of the element in the array of referenced objects in the property of the parent object.

[0096] Figure 15 illustrates how to access binary data. As shown in Figure 15, binary data is stored in a buffer object. In other words, the buffer object contains information for accessing the binary data (e.g., a URI (Uniform Resource Identifier)). In a JSON format file, as shown in Figure 15, objects such as meshes, cameras, and skins can access their buffer objects via accessor objects and bufferView objects.

[0097] In other words, for objects such as meshes, cameras, and skins, the accessor object to be referenced is specified. Figure 16 shows an example of how a mesh object is described in a JSON format file. For example, as shown in Figure 16, in a mesh object, vertex attributes such as NORMAL, POSITION, TANGENT, and TEXCORD_0 are defined as keys, and for each attribute, the accessor object to be referenced is specified as the value.

[0098] Figure 17 shows the relationship between buffer objects, buffer view objects, and accessor objects. Figure 18 shows an example of how these objects are described in a JSON format file.

[0099] In Figure 17, the buffer object 241 is an object that stores information (such as a URI) for accessing the actual binary data, and information indicating the data length (e.g., byte length) of that binary data. Figure 18A shows an example of the description of the buffer object 241. The "bytelength":102040 shown in Figure 18A indicates that the byte length of the buffer object 241 is 102040 bytes, as shown in Figure 17. Also, the "uri":duck.bin shown in Figure 18A indicates that the URI of the buffer object 241 is "duck.bin", as shown in Figure 17.

[0100] In Figure 17, the buffer view object 242 is an object that stores information about a subset of the binary data specified in the buffer object 241 (i.e., information about a portion of the buffer object 241). Figure 18B shows an example of how the buffer view object 242 is described. As shown in Figure 17 and Figure 18B, the buffer view object 242 stores information such as the identification information of the buffer object 241 to which the buffer view object 242 belongs, an offset (e.g., byte offset) indicating the position of the buffer view object 242 within the buffer object 241, and a length (e.g., byte length) indicating the data length (e.g., byte length) of the buffer view object 242.

[0101] As shown in Figure 18B, when multiple buffer view objects exist, information is written for each buffer view object (i.e., for each subset area). For example, the information shown at the top of Figure 18B, such as "buffer":0, "bytelength":25272, and "byteOffset":0, is the information of the first buffer view object 242 (bufferView[0]) shown within buffer object 241 in Figure 17. Similarly, the information shown at the bottom of Figure 18B, such as "buffer":0, "bytelength":76768, and "byteOffset":25272, is the information of the second buffer view object 242 (bufferView[1]) shown within buffer object 241 in Figure 17.

[0102] The "buffer":0 of ​​the first buffer view object 242 (bufferView[0]) shown in Figure 18B indicates that the identifier information of the buffer object 241 to which the buffer view object 242 (bufferView[0]) belongs is "0" (Buffer[0]), as shown in Figure 17. Furthermore, "bytelength":25272 indicates that the byte length of the buffer view object 242 (bufferView[0]) is 25272 bytes. Additionally, "byteOffset":0 indicates that the byte offset of the buffer view object 242 (bufferView[0]) is 0 bytes.

[0103] The "buffer":0 of ​​the second buffer view object 242 (bufferView[1]) shown in Figure 18B indicates that the identifier information of the buffer object 241 to which the buffer view object 242 (bufferView[0]) belongs is "0" (Buffer[0]), as shown in Figure 17. Furthermore, "bytelength":76768 indicates that the byte length of the buffer view object 242 (bufferView[0]) is 76768 bytes. Additionally, "byteOffset":25272 indicates that the byte offset of the buffer view object 242 (bufferView[0]) is 25272 bytes.

[0104] In Figure 17, the accessor object 243 is an object that stores information about how to interpret the data in the buffer view object 242. Figure 18C shows an example of how the accessor object 243 is described. As shown in Figure 17 and Figure 18C, the accessor object 243 stores information such as the identification information of the buffer view object 242 to which the accessor object 243 belongs, the offset (e.g., byte offset) indicating the position of the buffer view object 242 within the buffer object 241, the component type of the buffer view object 242, the number of data items stored in the buffer view object 242, and the type of data stored in the buffer view object 242. This information is described for each buffer view object.

[0105] In the example C in Figure 18, information such as "bufferView":0, "byteOffset":0, "componentType":5126, "count":2106, and "type":"VEC3" is shown. "bufferView":0 indicates that the identifier information of the buffer view object 242 to which the accessor object 243 belongs is "0" (bufferView[0]), as shown in Figure 17. Also, "byteOffset":0 indicates that the byte offset of the buffer view object 242 (bufferView[0]) is 0 bytes. Furthermore, "componentType":5126 indicates that the component type is of type FLOAT (OpenGL macro constant). Finally, "count":2106 indicates that there are 2106 data items stored in the buffer view object 242 (bufferView[0]). Furthermore, "type":"VEC3" indicates that the data (of type) stored in the buffer view object 242 (bufferView[0]) is a three-dimensional vector.

[0106] Access to data other than images is defined by a reference to this accessor object 243 (by specifying the accessor's index).

[0107] Next, we will explain how to specify a 3D object for a point cloud in a scene description (JSON format file) that conforms to glTF 2.0. A point cloud is 3D content that represents a three-dimensional structure (an object with a three-dimensional shape) as a collection of many points. The data of a point cloud consists of positional information (also called geometry) and attribute information (also called attributes) for each point. Attributes can include any information. For example, the attribute may include color information, reflectivity information, normal information, etc. for each point. In this way, point clouds have a relatively simple data structure and can represent any three-dimensional structure with sufficient accuracy by using a sufficiently large number of points.

[0108] When a point cloud does not change over time (also referred to as being static), 3D objects are specified using the glTF2.0 mesh.primitives object. Figure 19 shows an example of the object configuration in a scene description when the point cloud is static. Figure 20 shows an example of how that scene description is written.

[0109] As shown in Figure 20, the mode of the primitives object is set to 0, indicating that the data is treated as a point in a point cloud. As shown in Figures 19 and 20, the position property of the attributes object in mesh.primitives specifies an accessor to a buffer that stores the position information of the points. Similarly, the color property of the attributes object specifies an accessor to a buffer that stores the color information of the points. There may be only one buffer and one buffer view (the data may be stored in a single file).

[0110] Next, we will explain how to extend such scene description objects. Each object in glTF2.0 can store newly defined objects within an extension object. Figure 21 shows an example of how to specify a newly defined object (ExtensionExample). As shown in Figure 21, when using a newly defined extension, the extension object name (ExtensionExample in the example in Figure 21) is written in "extensionUsed" and "extensionRequired". This indicates that this extension is an extension that will be used or an extension that is required for loading.

[0111] <Client Processing> Next, we will explain the processing of the client device in MPEG-I Scene Description. The client device obtains the scene description, obtains 3D object data based on that scene description, and generates a display image using that scene description and 3D object data.

[0112] As described in Non-Patent Document 2, the client device performs processing using a presentation engine, media access function, etc. For example, as shown in Figure 22, the presentation engine 251 of the client device 250 acquires the initial value of the scene description and information for updating that scene description (hereinafter also referred to as update information), and generates a scene description for the processing target time. The presentation engine 251 then analyzes the scene description and identifies the media (video, audio, etc.) to be played. The presentation engine 251 then requests the media access function 252 to acquire that media via the Media Access API (Application Program Interface). The presentation engine 251 also sets up pipeline processing and specifies buffers, etc.

[0113] The media access function 252 retrieves various media data requested by the presentation engine 251 from the cloud, local storage, etc. The media access function 252 supplies the retrieved media data (encoded data) to the pipeline 253.

[0114] The pipeline 253 decodes various data (encoded data) from the supplied media through pipeline processing and supplies the decoded results to the buffer 254. The buffer 254 holds the various data from the supplied media.

[0115] The presentation engine 251 performs rendering and other operations using various media data held in the buffer 254.

[0116] <Application of Timed Media> In recent years, as shown in Non-Patent Document 2, for example, the application of timed media as 3D object content by extending glTF2.0 in MPEG-I Scene Description has been considered. Timed media is media data that changes along the time axis, such as moving images in two-dimensional images.

[0117] glTF could only be used with still image data as media data (3D object content). In other words, glTF did not support moving image media data. When 3D objects were to be moved, animation (a method of switching between still images along a time axis) was used.

[0118] In MPEG-I Scene Description, glTF 2.0 is applied, JSON format files are used as scene descriptions, and further extensions to glTF are being considered to allow the handling of timed media (e.g., video data) as media data. The following extensions are being considered to handle timed media, for example:

[0119] Figure 23 illustrates an extension for handling timed media. In the example in Figure 23, the MPEG media object (MPEG_media) is a glTF extension and is an object that specifies attributes of MPEG media such as video data, such as uri, track, renderingRate, and startTime.

[0120] Furthermore, as shown in Figure 23, an MPEG texture video object (MPEG_texture_video) is provided as an extension object of the texture object. This MPEG texture video object stores information about the accessor corresponding to the buffer object being accessed. In other words, the MPEG texture video object is an object that specifies the index of the accessor corresponding to the buffer in which the texture media specified by the MPEG media object (MPEG_media) is decoded and stored.

[0121] Figure 24 shows an example of how to describe an MPEG media object (MPEG_media) and an MPEG texture video object (MPEG_texture_video) in a scene description to illustrate extensions for handling timed media. In the example in Figure 24, the second line from the top shows that an MPEG texture video object (MPEG_texture_video) is set as an extension object (extensions) for a texture object (texture), as shown below. The value of that MPEG video texture object is specified as the index of the accessor (in this example, "2").

[0122] "texture":[{"sampler":0, "source":1, "extensions":{"MPEG_texture_video ":"accessor":2}}],

[0123] Furthermore, in the example shown in Figure 24, the MPEG media object (MPEG_media) is set as a glTF extension object (extensions) in lines 7 through 16 from the top, as shown below. Various information about the MPEG media object, such as its encoding and URI, is stored as the value of that MPEG media object.

[0124] "MPEG_media":{ "media":[ {"name":"source_1", "renderingRate":30.0, "startTime":9.0, "timeOffset":0.0, "loop":"true", "controls":"false", "alternatives":[{"mimeType":"video / mp4;codecs=\"avc1.42E01E\"", "uri":"video1.mp4", "tracks":[{"track":""#track_ID=1"}]}]} ]}

[0125] Furthermore, each frame data is decoded and sequentially stored in a buffer, but since its position and other properties change, the scene description includes a mechanism to store this changing information so that the renderer can read the data. For example, as shown in Figure 23, an MPEG buffer circular object (MPEG_buffer_circular) is provided as an extension object (extension) of the buffer object (buffer). This MPEG buffer circular object stores information for dynamically storing data within the buffer object. For example, information such as the data length of the buffer header (bufferHeader) and the number of frames is stored in this MPEG buffer circular object. The buffer header stores information such as the index, the timestamp of the frame data to be stored, and the data length.

[0126] Furthermore, as shown in Figure 23, an MPEG accessor timed object (MPEG_timed_accessor) is provided as an extension object (extension) of the accessor object. In this case, since the media data is video, the buffer view object (bufferView) referenced in the time direction may change (its position may fluctuate). Therefore, information indicating the referenced buffer view object is stored in this MPEG accessor timed object. For example, the MPEG accessor timed object stores information indicating a reference to the buffer view object (bufferView) in which a timedAccessor information header is described. The timedAccessor information header is, for example, header information that stores information within the dynamically changing accessor object and buffer view object.

[0127] Figure 25 shows an example of how to describe an MPEG buffer circular object (MPEG_buffer_circular) and an MPEG accessor timed object (MPEG_accessor_timed) in a scene description to explain extensions for handling timed media. In the example in Figure 25, the MPEG accessor timed object (MPEG_accessor_timed) is set as an extension object (extensions) of the accessor object (accessors) on the fifth line from the top, as shown below. The values ​​of the MPEG accessor timed object are specified as parameters and their values, such as the index of the buffer view object ("1" in this example), the update rate (updataRate), and immutable information.

[0128] "MPEG_accessor_timed":{"bufferView":1, "updateRate":25.0, "immutable":1,"}

[0129] Furthermore, in the example shown in Figure 25, on the 13th line from the top, an MPEG buffer circular object (MPEG_buffer_circular) is set as an extension object (extensions) of the buffer object (buffer), as shown below. And, as the value of that MPEG buffer circular object, parameters such as buffer frame count (count), header length (headerLength), and update rate (updataRate) and their values ​​are specified.

[0130] "MPEG_buffer_circular":{"count":5, "headerLength":12, "updateRate":25.0}

[0131] Figure 26 illustrates extensions for handling timed media. Figure 26 shows examples of the relationships between MPEG accessor timed objects and MPEG buffer circular objects, as well as accessor objects, buffer view objects, and buffer objects.

[0132] As mentioned above, the MPEG buffer circular object of a buffer object stores information necessary for storing time-varying data in the buffer area indicated by the buffer object, such as the buffer frame count (count), header length (headerLength), and update rate (updataRate). In addition, the buffer header (bufferHeader), which is the header of that buffer area, stores parameters such as the index (idex), timestamp (timestamp), and data length (length).

[0133] As mentioned above, the MPEG accessor timed object of an accessor object stores information about the buffer view object it references, such as the buffer view object's index (bufferView), update rate (updataRate), and immutable information. This MPEG accessor timed object also stores information about the buffer view object that stores the referenced timed accessor information header. The timed accessor information header may contain the timestamp delta (timestamp_delta), update data for the accessor object, update data for the buffer view object, and so on.

[0134] <Client Processing When Using MPEG_texture_video> A scene description is spatial placement information for arranging one or more 3D objects in 3D space. This scene description can be updated along the time axis. In other words, the placement of 3D objects can be updated as time progresses. The client processing performed on the client device in this case is described below.

[0135] Figure 27 shows an example of the main configuration of the client device regarding client processing, and Figure 28 is a flowchart showing an example of the flow of that client processing. As shown in Figure 27, the client device has a presentation engine (PresentationEngine (hereinafter also referred to as PE)) 251, a media access function (MediaAccessFunction (hereinafter also referred to as MAF)) 252, a pipeline 253, and a buffer 254. The presentation engine (PE) 251 has a glTF analysis unit 263 and a rendering unit 264.

[0136] The presentation engine (PE) 251 instructs the media access function 252 to acquire the media, retrieves the data via the buffer 254, and performs display-related processing. Specifically, the processing proceeds as follows:

[0137] When client processing begins, the glTF analysis unit 263 of the presentation engine (PE) 251 starts PE processing as shown in the example in Figure 28, and in step S21, it obtains the SD(glTF) file 262, which is a scene description file, and parses its scene description.

[0138] In step S22, the glTF analysis unit 263 confirms the media associated with the 3D object (texture), the buffer in which the media will be stored after processing, and the accessor. In step S23, the glTF analysis unit 263 notifies the media access function 252 of this information as a file acquisition request.

[0139] The Media Access Function (MAF) 252 starts MAF processing as shown in the example in Figure 28, and in step S11, it receives a notification. In step S12, the Media Access Function 252 obtains the media (3D object file (mp4)) based on the notification.

[0140] In step S13, the media access function 252 decodes the acquired media (3D object file (mp4)). In step S14, the media access function 252 stores the data of the decoded media in the buffer 254 based on a notification from the presentation engine (PE 251).

[0141] In step S24, the rendering processing unit 264 of the presentation engine 251 reads (acquires) the data from the buffer 254 at an appropriate timing. In step S25, the rendering processing unit 264 uses the acquired data to perform rendering and generate a display image.

[0142] The media access function 252 performs the processes in steps S13 and S14 repeatedly for each time point (each frame). The rendering processing unit 264 of the presentation engine 251 also performs the processes in steps S24 and S25 repeatedly for each time point (each frame). Once processing is complete for all frames, the media access function 252 terminates the MAF processing, and the presentation engine 251 terminates the PE processing. In other words, the client processing ends.

[0143] Non-Patent Document 2 describes the MPEG-I Scene Description, which is based on glTF 2.0 as described above and includes extensions for handling MPEG media content such as time-varying audio and video. Non-Patent Document 4 also describes the standardization of compression transmission technology for haptics media, which compresses tactile information, in addition to audio and video media, which are components of 2D video content and 3DoF / 6DoF video content.

[0144] In parallel with the standardization of compression transmission technology for haptic media, as described in Non-Patent Document 3, for example, the MPEG-I Scene Description 2nd Edition includes technical extensions for handling haptic media in 3D space and extensions for interaction technology. Haptic media is envisioned to have two types of models: a synchronous model that plays (vibrates) in sync with audio and video media, and an interactive model (touching, moving, bumping, etc.) between the viewer user and 3D video objects.

[0145] Furthermore, spatial haptics are realized by defining haptic information (static data, timed media) and mapping it to the mesh data (nodes) of 3D objects as surface information (MPEG_haptic_material), similar to texture information.

[0146] Furthermore, a basic structure is defined that maps the user avatar viewing the 3D scene to nodes, and it is also possible to associate haptics devices with the nodes of this avatar. To enhance the sense of realism, various haptics presentation devices equipped with numerous vibrators and actuators can be attached to various parts of the user's body, and haptics can be presented simultaneously.

[0147] <Method 1-9> For example, as shown in the top row of the table in Figure 29, haptics-related information may be transmitted using such a scene description (Method 1-9). For example, the encoding device may include a scene description generation unit that generates a scene description and stores haptics-related information about haptics in the generated scene description, and a haptics encoding unit that encodes source haptics data, which is haptics data at the source.

[0148] Furthermore, the encoding method performed by the encoding device may include generating a scene description, storing haptics-related information about haptics in the generated scene description, and encoding source haptics data, which is haptics data at the source.

[0149] Alternatively, the program may cause the computer to perform a process that includes generating a scene description, storing haptics-related information about haptics in the generated scene description, and encoding source haptics data, which is haptics data at the source.

[0150] Furthermore, the decoding device may further include a scene description acquisition unit that acquires a scene description containing haptics-related information relating to haptics. Then, a receptor haptics data generation unit may generate receptor haptics data based on the haptics-related information contained in the acquired scene description.

[0151] By doing so, haptics-related information can be transmitted using scene descriptions. Therefore, a wider variety of haptics experiences can be provided using this haptics-related information.

[0152] Furthermore, this haptics-related information may include any type of information. For example, it may include one or more of the information shown in Figure 30A. For example, the haptics-related information may include information that defines haptic objects within the scene. For example, the object definition shown in Figure 30A may be stored in the MPEG_haptic of the scene description. As shown in Figure 30A, this object definition may define Name, Haptic Source, Haptic Listener, Haptic transfer, etc.

[0153] Furthermore, the encoding device may further include a scene description supply unit that supplies the generated scene descriptions.

[0154] <Method 1-9-1> When Method 1-9 is applied, information regarding haptic propagation characteristics may be transmitted using a scene description, as shown in the second row from the top of the table in Figure 29 (Method 1-9-1). For example, the haptic-related information may include information regarding haptic propagation characteristics that indicate the propagation characteristics of haptic data by the haptic propagation medium that constitutes the propagation path of source haptic data from the source to the receptor. For example, a Haptic Transfer definition as shown in Figure 31B may be stored in MPEG_haptic.

[0155] The information regarding this haptics propagation characteristic may include any information. For example, it may include one or more of the settings for each parameter shown in Figure 31B. For example, the information regarding this haptics propagation characteristic may include identification information for the haptics propagation characteristic. For example, as shown in Figure 31B, id may be stored in MPEG_haptic. The information regarding this haptics propagation characteristic may also include information regarding the operation settings when the renderer is not supported. For example, as shown in Figure 31B, Bypass may be stored in MPEG_haptic. The information regarding this haptics propagation characteristic may also include information regarding the type of haptics propagation characteristic. For example, as shown in Figure 31B, customProperty_enable may be stored in MPEG_haptic. The information regarding this haptics propagation characteristic may also include information regarding the property definition of the haptics propagation characteristic. For example, as shown in Figure 31B, Properties may be stored in MPEG_haptic. Furthermore, this information regarding haptic propagation characteristics may also include information regarding the propagation delay of haptic data. For example, as shown in Figure 31B, predelay may be stored in MPEG_haptic.

[0156] <Method 1-9-2> When Method 1-9 is applied, source information may be transmitted using a scene description, as shown in the third row from the top of the table in Figure 29 (Method 1-9-2). For example, haptics-related information may include source information. For example, a Haptic Source definition, as shown in Figure 30B, may be stored in MPEG_haptic.

[0157] The information regarding this source may include any information. For example, it may include one or more of the parameter settings shown in Figure 30B. For example, the information regarding this source may include source identification information. For example, Id may be stored in MPEG_haptic as shown in Figure 30B. The information regarding this source may also include information about the type of source. For example, Type may be stored in MPEG_haptic as shown in Figure 30B. The information regarding this source may also include information about the source's pre-gain. For example, Pregain may be stored in MPEG_haptic as shown in Figure 30B. The information regarding this source may also include information about the response during audio playback. For example, PlaybackSpeedSync may be stored in MPEG_haptic as shown in Figure 30B. The information regarding this source may also include information about the specification of the attenuation function. For example, Attenuation may be stored in MPEG_haptic as shown in Figure 30B. The information regarding this source may also include information about the parameters applied to the attenuation function. For example, as shown in Figure 30B, Attenuation Parameters may be stored in MPEG_haptic. Furthermore, this source information may include information about the distance at which the gain becomes 1. For example, as shown in Figure 30B, referenceDistance may be stored in MPEG_haptic. Furthermore, this source information may include information about accessors. For example, as shown in Figure 30B, accessors may be stored in MPEG_haptic. Furthermore, this source information may include information about the presence or absence of air propagation. For example, as shown in Figure 30B, transferAir_enable may be stored in MPEG_haptic.

[0158] <Method 1-9-3> When Method 1-9 is applied, information about the receptor may be transmitted using a scene description, as shown in the bottom row of the table in Figure 29 (Method 1-9-3). For example, haptics-related information may include information about the receptor. For example, a Haptic Listener definition as shown in Figure 31A may be stored in MPEG_haptic.

[0159] The information about this receptor may include any information. For example, it may include one or more of the parameter settings shown in Figure 31A. For example, the information about this receptor may include receptor identification information. For example, the ID may be stored in MPEG_haptic as shown in Figure 31A.

[0160] Examples of Haptic Transfer propertyity definitions are shown in Figures 32A and 32B. An example of an extension of MPEG_haptic_material is shown in Figure 33. One or more of these parameter settings may be included in the information regarding haptic propagation properties (or haptic-related information).

[0161] <5. First Embodiment> <Reproduction Device> This technology can be applied to any device. Figure 34 is a block diagram showing an example of the configuration of a reproduction device, which is one embodiment of a decoding device to which this technology is applied. The reproduction device 300 shown in Figure 34 is a device that decodes and reproduces a bitstream of haptic data.

[0162] Note that Figure 34 shows the main components such as the processing unit and data flow, and does not necessarily represent everything. In other words, the playback device 300 may have processing units that are not shown as blocks in Figure 34, or processes and data flows that are not shown as arrows or other symbols in Figure 34.

[0163] As shown in Figure 34, the playback device 300 includes a haptics decoding unit 311, a haptics processing unit 312, a haptics presentation unit 313, an input unit 314, and a sensor unit 315. The haptics processing unit 312 also includes a haptics propagation path length derivation unit 321, a receptor haptics data generation unit 322, and a haptics propagation characteristic setting unit 323.

[0164] The haptics decoding unit 311 acquires a bitstream of haptics data, decodes it, and generates haptics data. The haptics decoding unit 311 supplies the generated haptics data to the receptor haptics data generation unit 322. The haptics processing unit 312 performs processing related to the haptics data. The haptics presentation unit 313 is a haptics device having actuators, etc., which acquires receptor haptics data supplied from the receptor haptics data generation unit 322 and outputs physical motion corresponding to that receptor haptics data. The input unit 314 receives input from the user or application, etc., and supplies it to the haptics processing unit 312. The sensor unit 315 observes the surroundings of the playback device 300 and detects desired parameters related to the external environment. The sensor unit 315 supplies the detected values ​​to the haptics processing unit 312.

[0165] Furthermore, the haptics propagation path length derivation unit 321 identifies the source location and the receptor location, identifies the haptics propagation path between them, and derives the length of the haptics propagation path. The haptics propagation path length derivation unit 321 supplies the derived haptics propagation path length to the receptor haptics data generation unit 322.

[0166] The receptor haptics data generation unit 322 acquires the haptics propagation path length, haptics propagation characteristics, etc., and uses them to convert source haptics data into receptor haptics data. The receptor haptics data generation unit 322 supplies the generated receptor haptics data to the haptics presentation unit 313.

[0167] The haptics propagation characteristics setting unit 323 sets the haptics propagation characteristics and supplies them to the receptor haptics data generation unit 322.

[0168] <Regeneration Process Flow 1> This technology can be applied to the regeneration device 300. For example, the method 1-1 described above can be applied to the regeneration device 300. An example of the regeneration process flow performed by the regeneration device 300 in that case will be explained with reference to the flowchart in Figure 35.

[0169] In this case, in step S301, the haptics decoding unit 311 decodes the bitstream of source haptics data. In step S302, the haptics propagation path length derivation unit 321 determines the source position and the receptor position, and derives the haptics propagation path length based on them. In step S303, the receptor haptics data generation unit 322 generates receptor haptics data using the source haptics data, the haptics propagation path length, and the spatial haptics propagation characteristics. In step S304, the haptics presentation unit 313 outputs the receptor haptics data.

[0170] By executing each process in this manner, it becomes possible to provide a wider variety of haptic experiences.

[0171] <Regeneration Process Flow 2> Methods 1-2 described above can also be applied to the regeneration device 300. An example of the regeneration process flow performed by the regeneration device 300 in this case will be explained with reference to the flowchart in Figure 36.

[0172] In this case, in step S321, the haptics decoding unit 311 decodes the bitstream of source haptics data. In step S322, the haptics propagation path length derivation unit 321 determines the source position and the receptor position, and derives the haptics propagation path length based on these and the object shape. In step S323, the receptor haptics data generation unit 322 generates receptor haptics data using the source haptics data, the haptics propagation path length, and the haptics propagation characteristics of the object. In step S324, the haptics presentation unit 313 outputs the receptor haptics data.

[0173] By executing each process in this manner, it becomes possible to provide a wider variety of haptic experiences.

[0174] <Regeneration Process Flow 3> Methods 1-3 described above can also be applied to the regeneration device 300. An example of the regeneration process flow performed by the regeneration device 300 in that case will be explained with reference to the flowchart in Figure 37.

[0175] In this case, in step S341, the haptics decoding unit 311 decodes the bitstream of source haptics data. In step S342, the haptics propagation path length derivation unit 321 determines the source position and the receptor position, and derives the haptics propagation path length based on them. In step S343, the haptics propagation characteristic setting unit 323 sets the haptics propagation characteristics. In step S344, the receptor haptics data generation unit 322 generates receptor haptics data using the source haptics data, the haptics propagation path length, and the haptics propagation characteristics. In step S345, the haptics presentation unit 313 outputs the receptor haptics data.

[0176] By executing each process in this manner, it becomes possible to provide a wider variety of haptic experiences.

[0177] <Regeneration Process Flow 4> Methods 1-4 described above can also be applied to the regeneration device 300. An example of the regeneration process flow performed by the regeneration device 300 in this case will be explained with reference to the flowchart in Figure 38.

[0178] In this case, in step S361, the haptics decoding unit 311 decodes the bitstream of source haptics data. In step S362, the haptics propagation path length derivation unit 321 determines the source position and the receptor position, and derives the haptics propagation path length based on them. In step S363, the sensor unit 315 senses the outside and generates sensor values. In step S364, the haptics propagation characteristic setting unit 323 sets the haptics propagation characteristics using the sensor values. In step S365, the haptics presentation unit 313 outputs receptor haptics data.

[0179] By executing each process in this manner, it becomes possible to provide a wider variety of haptic experiences.

[0180] <Regeneration Process Flow 5> Methods 1-5 described above can also be applied to the regeneration device 300. An example of the regeneration process flow performed by the regeneration device 300 in this case will be explained with reference to the flowchart in Figure 39.

[0181] In this case, in step S381, the haptics decoding unit 311 decodes the bitstream of source haptics data. In step S382, the haptics propagation path length derivation unit 321 determines the source position and the receptor position, and derives the haptics propagation path length based on them. In step S383, the receptor haptics data generation unit 322 generates receptor haptics data using the source haptics data, the haptics propagation path length, and the spatial haptics propagation characteristics. In step S384, the sensor unit 315 senses the outside and generates sensor values. In step S385, the receptor haptics data generation unit 322 updates the receptor haptics data using the sensor values. In step S386, the haptics presentation unit 313 outputs the updated receptor haptics data using the sensor values.

[0182] By executing each process in this manner, it becomes possible to provide a wider variety of haptic experiences.

[0183] <Regeneration Process Flow 6> Methods 1-6 described above can also be applied to the regeneration device 300. An example of the regeneration process flow performed by the regeneration device 300 in this case will be explained with reference to the flowchart in Figure 40.

[0184] In this case, in step S401, the haptics decoding unit 311 decodes the bitstream of source haptics data. In step S402, the haptics propagation path length derivation unit 321 determines the source position and the receptor position, and derives the haptics propagation path length based on them. In step S403, the receptor haptics data generation unit 322 generates receptor haptics data using the source haptics data, the haptics propagation path length, and the haptics propagation characteristics of the avatar. In step S404, the haptics presentation unit 313 outputs the receptor haptics data.

[0185] By executing each process in this manner, it becomes possible to provide a wider variety of haptic experiences.

[0186] <Regeneration Process Flow 7> Method 1-7 described above can also be applied to the regeneration device 300. An example of the regeneration process flow performed by the regeneration device 300 in that case will be explained with reference to the flowchart in Figure 41.

[0187] In this case, in step S421, the haptics decoding unit 311 decodes the bitstream of source haptics data. In step S422, the haptics propagation path length derivation unit 321 determines the source position and receptor position, and derives the haptics propagation path length based on them. In step S423, the receptor haptics data generation unit 322 generates a plurality of receptor haptics data using the source haptics data, the haptics propagation path length, and the haptics propagation characteristics. Then, in step S424, the receptor haptics data generation unit 322 mixes the plurality of receptor haptics data. In step S425, the haptics presentation unit 313 outputs the physical motion corresponding to the mixed receptor haptics data.

[0188] By executing each process in this manner, it becomes possible to provide a wider variety of haptic experiences.

[0189] <Regeneration Process Flow 8> An example of the regeneration process flow performed by the regeneration device 300 when method 1-8-1 is applied will be explained with reference to the flowchart in Figure 42.

[0190] In this case, in step S441, the haptics propagation characteristic setting unit 323 convolves multiple haptics propagation characteristics that constitute the haptics propagation path and derives a convolutional haptics propagation characteristic. In step S442, the haptics decoding unit 311 decodes the bitstream of source haptics data. In step S443, the haptics propagation path length derivation unit 321 determines the source position and the receptor position and derives the haptics propagation path length based on them. In step S444, the receptor haptics data generation unit 322 generates receptor haptics data using the source haptics data, the haptics propagation path length, and the convolutional haptics propagation characteristic. In step S445, the haptics presentation unit 313 outputs the physical motion corresponding to the mixed receptor haptics data.

[0191] By executing each process in this manner, it becomes possible to provide a wider variety of haptic experiences.

[0192] <Regeneration Process Flow 9> An example of the regeneration process flow performed by the regeneration device 300 when method 1-8-2 is applied will be explained with reference to the flowchart in Figure 43.

[0193] In this case, in step S461, the haptics decoding unit 311 decodes the bitstream of source haptics data. In step S462, the haptics propagation path length derivation unit 321 determines the source position and receptor position and derives the haptics propagation path length based on them. In step S463, the haptics propagation characteristic setting unit 323 convolves multiple haptics propagation characteristics according to the set haptics propagation path and derives the convolutional haptics propagation characteristics. In step S464, the receptor haptics data generation unit 322 generates receptor haptics data using the source haptics data, the haptics propagation path length, and the convolutional haptics propagation characteristics. In step S465, the haptics presentation unit 313 outputs the physical motion corresponding to the mixed receptor haptics data.

[0194] By executing each process in this manner, it becomes possible to provide a wider variety of haptic experiences.

[0195] <6. Second Embodiment> <File Generation Device> This technology can be applied to any device. Figure 44 is a block diagram showing an example of the configuration of a file generation device, which is one embodiment of an encoding device to which this technology is applied. The file generation device 500 shown in Figure 44 is a device that encodes haptic data and outputs its bitstream. The file generation device 500 is also a device that generates and outputs its scene description.

[0196] Note that Figure 44 shows the main components such as the processing unit and data flow, and does not necessarily represent everything. In other words, the file generation device 500 may have processing units that are not shown as blocks in Figure 44, or processes and data flows that are not shown as arrows or other symbols in Figure 44.

[0197] As shown in Figure 44, the file generation device 500 includes an SD generation unit 511, an SD encoding unit 512, a haptic encoding unit 513, a storage unit 514, and a supply unit 515.

[0198] The SD generation unit 511 generates a scene description using haptic data, etc., and supplies it to the SD encoding unit 512. The SD encoding unit 512 encodes the scene description and generates a bitstream. This may be stored in a container file or the like if necessary (a scene description file may be generated). The SD encoding unit 512 supplies the generated bitstream to the storage unit 514. The haptic encoding unit 513 encodes haptic data and generates a bitstream. This may be stored in a container file or the like if necessary (a content file may be generated). The haptic encoding unit 513 supplies the generated bitstream to the storage unit 514. The storage unit 514 stores the scene description bitstream (scene description file), the haptic data bitstream (content file), etc. The storage unit 514 supplies to the supply unit 515 at a predetermined timing or based on requests from other users or applications. The supply unit 515 supplies the bitstream of scene descriptions and haptic data (scene description files and content files) read from the storage unit 514 to other devices.

[0199] <File Generation Process Flow> For example, Method 1-9 of this technology can be applied to this file generation device 500. An example of the file generation process flow executed by the file generation device 500 in that case will be explained with reference to the flowchart in Figure 45.

[0200] When the file generation process starts, in step S501, the SD generation unit 511 generates a scene description and stores haptics-related information. In step S502, the SD encoding unit 512 encodes the scene description and generates a bitstream of the scene description (scene description file). In step S503, the storage unit 514 stores the bitstream of the scene description (scene description file). In step S504, the supply unit 515 reads the bitstream of the scene description (scene description file) and supplies it to another device.

[0201] Furthermore, in step S505, the haptics encoding unit 513 encodes the haptics data and generates a bitstream (content file) of the haptics data.

[0202] In step S506, the storage unit 514 stores the bitstream (content file) of the haptic data.

[0203] In step S507, the supply unit 515 reads the haptic data bitstream (content file) and supplies it to an external device.

[0204] By executing each process in this manner, it becomes possible to provide a wider variety of haptic experiences.

[0205] <Playback Device> This technology can be applied to any device. Figure 46 is a block diagram showing an example of the configuration of a playback device, which is one embodiment of a decoding device to which this technology is applied. The playback device 600 shown in Figure 46 corresponds to the file generation device 500 and is a device that decodes and plays back the bitstream of haptic data generated by the file generation device 500 using the scene description generated by the file generation device 500.

[0206] Note that Figure 46 shows the main components such as the processing unit and data flow, and does not necessarily represent everything. In other words, the playback device 600 may have processing units that are not shown as blocks in Figure 46, or processes and data flows that are not shown as arrows or other symbols in Figure 46.

[0207] As shown in Figure 46, the playback device 600 includes an MAF 611, a buffer 612, a PE 613, a sensor unit 614, an input unit 615, and a haptics display unit 616. The MAF 611 includes a content file acquisition unit 621 and a haptics decoding unit 622. The PE 613 includes an SD acquisition unit 631, an SD analysis unit 632, and a haptics processing unit 633.

[0208] The MAF 611 performs processing related to content acquisition and decoding. The MAF 611 supplies the processed data to the PE 613 via the buffer 612, etc. The PE 613 performs control and processing related to haptic data. The sensor unit 614, input unit 615, and haptic display unit 616 are processing units similar to the haptic display unit 313, input unit 314, and sensor unit 315.

[0209] The content file acquisition unit 621 acquires a bitstream of haptic data (content file) according to the control of the PE and supplies it to the haptic decoding unit 622. The haptic decoding unit 622 decodes the bitstream of haptic data (content file) and generates haptic data. The haptic decoding unit 622 supplies the haptic data to the haptic processing unit 633 via the buffer 612.

[0210] The SD acquisition unit 631 acquires and decodes the scene description bitstream (scene description file) to obtain the scene description. The SD acquisition unit 631 supplies the obtained scene description to the SD analysis unit 632. The SD analysis unit 632 analyzes the scene description and controls the content file acquisition unit 621 based on the analysis results. The SD analysis unit 632 also supplies the analysis results to the haptics processing unit 633. The configuration of the haptics processing unit 633 is the same as that of the haptics processing unit 312 in Figure 34.

[0211] <Playback Process Flow> An example of the playback process flow in this case will be explained with reference to the flowchart in Figure 47. When the playback process starts, in step S601, the SD acquisition unit 631 acquires and decodes the bitstream of the scene description (scene description file) that stores haptics-related information, and obtains the scene description. In step S602, the SD analysis unit 632 analyzes the scene description. In step S603, the content file acquisition unit 621 acquires the bitstream (content file) of each media based on the analysis result of the scene description. For example, the content file acquisition unit 621 acquires the bitstream of source haptics data as the bitstream of haptics data.

[0212] In step S604, the haptics decoding unit 622 decodes the bitstream of the source haptics data to obtain source haptics data. In step S605, the haptics processing unit 633 determines the source location and receptor location, and derives the haptics propagation path length based on them. In step S606, the haptics processing unit 633 sets the haptics propagation characteristics based on the haptics-related information of the scene description. In step S607, the haptics processing unit 633 generates receptor haptics data using the source haptics data, the haptics propagation path length, and the haptics propagation characteristics. In step S608, the haptics presentation unit 616 outputs the receptor haptics data.

[0213] By executing each process in this manner, it becomes possible to provide a wider variety of haptic experiences.

[0214] <7. Application Examples> <Method 1-10> When Method 1 is applied, the branching of the propagation path may be represented in the propagation path configuration information, as shown in the top row of the table in Figure 48 (Method 1-10).

[0215] For example, suppose the propagation path from the source to the receptor is composed of a single haptic propagation medium. As shown in the examples of Figure 49A and Figure 49B, when multiple propagation paths are composed of a common haptic propagation medium M, the receptor haptic data corresponding to each propagation path is derived using the haptic propagation characteristics of that haptic propagation medium M. In contrast, as shown in the examples of Figure 49C and Figure 49D, when multiple propagation paths are composed of different haptic propagation mediums M, the receptor haptic data corresponding to each propagation path is derived individually using the haptic propagation characteristics of each haptic propagation medium M.

[0216] Furthermore, as described above in Method 1-8, etc., when the propagation path from the source to the receptor is composed of multiple haptic propagation media, the receptor haptic data is derived by convolving the haptic propagation characteristics of each stage. In other words, even if a common haptic propagation medium is used for multiple propagation paths, the receptor haptic data for each propagation path is derived individually, as shown in the example in Figure 50.

[0217] In the example shown in Figure 50, the car engine is the haptic source, the surface of the avatar's hand or handle is the haptic listener A, and the surface of the avatar's back or car seat is the haptic listener B. The propagation path A from the source to the haptic listener A is composed of the car frame and handle, and the propagation path B from the source to the haptic listener B is composed of the car frame and car seat. The receptor haptic data for propagation path A is derived using haptic medium A, which is a convolution of the haptic propagation characteristics of the car frame and handle. The receptor haptic data for propagation path B is derived using haptic medium B, which is a convolution of the haptic propagation characteristics of the car frame and car seat. In other words, the receptor haptic data for each propagation path is derived individually.

[0218] Therefore, when multiple propagation paths have common parts (i.e., when a common haptic propagation medium is used for each propagation path), redundant parts arise in the derivation of receptor haptic data for each propagation path, as described above, which could unnecessarily increase the processing load.

[0219] Therefore, as in method 1-10 described above, the propagation path configuration information is made to represent the branching of the propagation path. For example, in a decoding device, the receptor haptics data generation unit may further generate receptor haptics data using propagation path configuration information that shows the configuration of the propagation path including branching.

[0220] For example, if the propagation path from source S to receptor L has the configuration shown in Figure 51, this is described as branching from a common haptic propagation medium M into multiple haptic propagation mediums M. Then, haptic data is derived for each haptic propagation medium M, and the intermediate results are stored and used while deriving receptor haptic data for each propagation path. In other words, the processing is divided into a part that calculates using the transfer function of the common haptic propagation medium M and a part that calculates using the transfer function of the individual haptic propagation medium M.

[0221] For example, in a decoding device, a receptor haptics data generation unit may derive receptor haptics data based on propagation path configuration information that includes a first propagation path from the source to a first receptor via a first haptics propagation medium and a second haptics propagation medium, and a second propagation path from the source to a second receptor via a first haptics propagation medium and a third haptics propagation medium. In this case, the receptor haptics data generation unit may generate first haptics data for the first and second propagation paths using source haptics data, a first propagation path length which is the path length of the first haptics propagation medium, and first haptics propagation characteristics provided by the first haptics propagation medium. Furthermore, the receptor haptics data generation unit may generate first receptor haptics data sensed by the first receptor using the first haptics data, a second propagation path length which is the path length of the second haptics propagation medium, and a second haptics propagation characteristic of the second haptics propagation medium for the first propagation path. Furthermore, the receptor haptics data generation unit may generate second receptor haptics data sensed by the second receptor using the first haptics data, a third propagation path length which is the path length of the third haptics propagation medium, and a third haptics propagation characteristic of the third haptics propagation medium for the second propagation path.

[0222] For example, in the case shown in Figure 50, as shown in Figure 52, haptic data is derived and stored after propagation of the car frame (Haptic Medium 1), which is the haptic propagation medium, from the car engine (Haptic Source), which is the source S. Then, using the haptic data after propagation of the car frame, haptic data is derived after propagation of the handle (Haptic Medium 2A), which is the haptic propagation medium, i.e., receptor haptic data sensed at the avatar's hand or handle leaf surface (Haptic Listener A), which is the receptor. Furthermore, using the haptic data after propagation of the car frame, haptic data is derived after propagation of the car sheet (Haptic Medium 2B), which is the haptic propagation medium, i.e., receptor haptic data sensed at the avatar's back or car sheet leaf surface (Haptic Listener B), which is the receptor.

[0223] By doing so, it is possible to suppress the execution of calculations related to the propagation of the common part, the Kerr frame (Haptic Medium 1), multiple times. In other words, redundant processing can be reduced, and the increase in processing load can be suppressed. This can suppress increases in processing time, power consumption, and cost.

[0224] Note that there can be any number of branching events, and as shown in the example in Figure 53, branching may occur two or more times. In the example in Figure 53, the propagation destination of "Haptic Medium 2B," which branches off from "Haptic Medium 1," is further branched into "Haptic Medium 3B" and "Haptic Medium 3C."

[0225] Furthermore, the branching of the haptic data propagation destination can be set at any point along the propagation path. As shown in the example in Figure 54, the propagation destination may branch from the source (Haptic Source). Also, the number of branching points to each receptor does not have to be the same. As shown in the example in Figure 54, the number of branching points to receptor A (Haptic Listener A) may be 1, while the number of branching points to receptor B (Haptic Listener B) and the number of branching points to receptor C (Haptic Listener C) may be 2.

[0226] Furthermore, propagation paths from different sources may pass through a common haptic propagation medium. Moreover, these propagation paths may reach a common receptor. For example, if the propagation path has the configuration shown in Figure 55, receptor B (Haptic Listener B) outputs the sum of haptic data output from source A and sensed via haptic medium 1A and haptic medium 2B, and haptic data output from source B and sensed via haptic medium 1B and haptic medium 2B.

[0227] Furthermore, this propagation path configuration information may be transmitted. For example, the information shown in Figure 56 may be transmitted as this propagation path configuration information. In other words, the information shown in Figure 56 may be included in the transmitted propagation path configuration information. For example, this propagation path configuration information may be transmitted from the encoding device to the decoding device. That is, the encoding device may generate this propagation path configuration information and transmit it to the decoding device, and the decoding device may use this propagation path configuration information to derive receptor haptic data.

[0228] For example, as part of this propagation path configuration information, the parameter "hapticMediums" may be transmitted, as shown in Figure 56. This "hapticMediums" consists of an array of haptic propagation media (MPEG_spatial_haptics.Medium) and provides a list of haptic propagation media at the glTF file level to enable support for spatial haptic data. Note that this "hapticMediums" is a required parameter, and its initial value is "Not Applicable (N / A)".

[0229] Furthermore, as part of this propagation path configuration information, the parameter "hapticSources" is transmitted, as shown in Figure 56. This "hapticSources" consists of an array of sources (MPEG_spatial_haptics.Source) and provides a list of sources (Haptic Sources) at the glTF file level to enable support for spatial haptics data. Note that this "hapticSources" is a required parameter, and its initial value is "Not applicable (N / A)".

[0230] Furthermore, as part of this propagation path configuration information, the parameter "hapticListeners" is transmitted, as shown in Figure 56. This "hapticListeners" consists of an array of receptors (MPEG_spatial_haptics.Listener) and provides a list of receptors (Haptic Listeners) at the glTF file level to enable support for spatial haptic data. Note that this "hapticListeners" is a required parameter, and its initial value is "Not applicable (N / A)".

[0231] Furthermore, as part of this propagation path configuration information, the parameter "hapticPipeline" is transmitted, as shown in Figure 56. This "hapticPipeline" is composed of an array of haptic pipelines (MPEG_spatial_haptics.Pipeline) and provides a list of haptic pipelines (hapticPipeline) at the glTF file level to enable support for spatial haptic data. This haptic pipeline (hapticPipeline) shows the propagation path from the source to the receptor as an array of haptic propagation media that constitute that propagation path. The initial value of this "hapticPipeline" is "Not applicable (N / A)".

[0232] The sequence of haptic pipelines (MPEG_spatial_haptics.Pipeline) can be described as information grouped by propagation path configuration units. In other words, for the sources (hapticSources), haptic propagation media (hapticMediums), and receptors (hapticListeners) that constitute the propagation path, an index of the sequence of haptic pipelines (MPEG_spatial_haptics.Pipeline), which is information indicating that they belong to the same group, is assigned.

[0233] If the structure of the scene can change over time (i.e., if the source, haptic propagation medium, receptors, etc. can change over time), the information in "hapticPipeline" may be used as an initial value. For example, the parameter "isInitial" may be set for "hapticPipeline" as shown in Figure 57. This "isInitial" is of data type "Boolean" and indicates whether "hapticPipeline" is initial information or not. In other words, this "isInitial" specifies whether "hapticPipeline" may be modified by interactions or other information. For example, if this "isInitial" is true, "hapticPipeline" indicates the initial state of the propagation path. The initial value of this "isInitial" is false.

[0234] Furthermore, the parameter "pipelineInformation" may be set for this "hapticPipeline," as shown in Figure 57. This "pipelineInformation" indicates how the rendered haptic signal depends on the haptic data associated with the source, the haptic propagation characteristics of the haptic propagation medium, and the receptor information. The processing order through the medium from the source to the receptor is described by the following comma-separated indices: "M#" indicates the index of the haptic propagation medium (hapticMediums), "S#" indicates the index of the source (hapticSources), and "L#" indicates the index of the receptor (hapticListeners). Note that this "pipelineInformation" is mandatory information, its data type is "string," and its initial value is "Not Applicable (N / A)."

[0235] For example, suppose the propagation path has the configuration shown in Figure 58. That is, the car engine is the haptic source, the car frame is the haptic medium 1, the steering wheel is the haptic medium 2A, the car seat is the haptic medium 2B, the surface of the steering wheel is the receptor (Haptic Listener A), and the surface of the car seat is the receptor (Haptic Listener B). Furthermore, the identification information for the car engine (Haptic Source) is "S#1" (source Index = 1), the identification information for the car frame (Haptic Medium 1) is "M#1" (Medium Index = 1), the identification information for the steering wheel (Haptic Medium 2A) is "M#2" (Medium Index = 2), the identification information for the car seat (Haptic Medium 2B) is "M#3" (Medium Index = 3), the identification information for the steering wheel surface (Haptic Listener A) is "L#1" (Listener Index = 1), and the identification information for the car seat surface (Haptic Listener B) is "L#2" (Listener Index = 2).

[0236] In this case, the haptic pipeline may be described in "pipelineInformation" in the format shown in Figure 59A. In the example of Figure 59A, the propagation path from the car engine (Haptic Source) to the steering wheel surface (Haptic Listener A) is described as "S#1, M#1, M#2, L#1". The propagation path from the car engine (Haptic Source) to the car seat surface (Haptic Listener B) is described as "S#1, M#1, M#3, L#2". Thus, each propagation path may be described in an array.

[0237] Furthermore, in "pipelineInformation," the haptic pipeline may be described in the format shown in Figure 59B. In the example in Figure 59A, the propagation path from the car engine (Haptic Source) to the steering wheel surface (Haptic Listener A) and the propagation path from the car engine (Haptic Source) to the car seat surface (Haptic Listener B) are described as "(S#1,M#1,M#2,L#1) | (S#1,M#1,M#3,L#2)" or "(S#1,M#1), (M#2,L#1) | (M#3,L#2)". In other words, multiple propagation paths may be combined and represented in a single array.

[0238] Furthermore, by assigning a unique ID (e.g., SpatialHapticID) to the parameters "hapticMediums," "hapticSources," and "hapticListeners" shown in Figure 56, which can be identified even with different parameters, the parameter "pipelineInformation" shown in Figure 57 may be described using a unique ID instead of an index indicating each array.

[0239] <Flow of Regeneration Process> This technology (Method 1-10) can be applied to any device. For example, this technology (Method 1-10) may be applied to the regeneration device 300 shown in Figure 34. In that case, the regeneration device 300 has the same configuration as the example in Figure 34. An example of the flow of the regeneration process performed by the regeneration device 300 in this case will be explained with reference to the flowchart in Figure 60.

[0240] In this case, in step S701, the haptics decoding unit 311 decodes the bitstream of source haptics data. In step S702, the haptics propagation path length derivation unit 321 determines the source location and receptor location, and derives the haptics propagation path length based on these and the propagation path configuration information. In step S703, the haptics propagation characteristic setting unit 323 sets the haptics propagation characteristics based on the source location, receptor location, and propagation path configuration information. In step S704, the receptor haptics data generation unit 322 generates receptor haptics data using the source haptics data, the haptics propagation path length, and the haptics propagation characteristics. In step S705, the haptics presentation unit 313 outputs the receptor haptics data.

[0241] By executing each process in this manner, a wider variety of haptic experiences can be provided. Furthermore, an increase in processing load can be suppressed.

[0242] Furthermore, the propagation path configuration information may be transmitted from the encoding device to the decoding device. In other words, this technology (Method 1-10) may be applied to the file generation device 500 shown in Figure 44. In that case, the file generation device 500 may store its propagation path configuration information as haptics-related information in the scene description. Therefore, in this case as well, the file generation device 500 has the same configuration as the example in Figure 44. The file generation device 500 then executes the file generation process in the same flow as the flowchart in Figure 45. In other words, this technology (Method 1-10) may be applied to the file generation process shown in Figure 45.

[0243] Furthermore, this technology (Method 1-10) may be applied to the playback device 600 shown in Figure 46. In that case, the playback device 600 may acquire a scene description including haptics-related information including propagation path configuration information, and perform processing related to the generation of receptor haptics data based on that scene description (i.e., propagation path configuration information). Therefore, in this case as well, the playback device 600 has the same configuration as in Figure 46. The playback device 600 then performs the playback process in the same flow as the flowchart in Figure 47. In other words, this technology (Method 1-10) may be applied to the playback process shown in Figure 47.

[0244] <Method 1-11> Conventionally, it has been difficult to emphasize or reduce a particular path in haptic data that arrives via various paths. Therefore, there is a risk that it may interfere when there is a lot of noise. So, when Method 1 is applied, the use of the haptic propagation medium (Medium) in the propagation path may be controlled as shown in the second row from the top of the table in Figure 48 (Method 1-11).

[0245] For example, in a decoding device, the receptor haptics data generation unit may further generate receptor haptics data using propagation path control information that controls the use of the haptics propagation medium.

[0246] This makes it easier to switch whether or not a haptic propagation medium propagates haptic data. Therefore, it becomes easier to create propagation paths (receptor haptic data).

[0247] For example, this propagation path control information may control whether the haptic propagation medium is turned on or off (i.e., whether or not it is used as a transmission path). For example, in a decoding device, the propagation path control information may indicate whether or not to use the haptic propagation medium. Then, a receptor haptic data generation unit may generate receptor haptic data for propagation paths that are configured with the haptic propagation medium that has been indicated to be used by the propagation path control information.

[0248] Furthermore, this propagation path control information may be used to control the priority of haptic propagation media (i.e., whether to use them preferentially as transmission paths). For example, in a decoding device, the propagation path control information may indicate the priority of using haptic propagation media. Then, the receptor haptic data generation unit may generate receptor haptic data for propagation paths composed of haptic propagation media that have been given a higher priority in the propagation path control information.

[0249] Furthermore, the propagation path control information may control the threshold for the difference in levels of the receptor haptics data. For example, in a decoding device, the propagation path control information may further indicate the threshold for the difference in levels of the receptor haptics data. If the level of the first receptor haptics data is greater than the level of the second receptor haptics data compared to the threshold indicated by the propagation path control information, the receptor haptics data generation unit may block the propagation path by not using the haptics propagation medium that constitutes the propagation path of the second receptor haptics data, and erase the second receptor haptics data.

[0250] Furthermore, the use of the source may be further controlled by this propagation path control information. For example, in a decoding device, the propagation path control information may control the use of the source. Then, a receptor haptics data generation unit may generate receptor haptics data corresponding to the source based on the propagation path control information.

[0251] For example, in a car racing video game, the vibrations from the engine of the car driven by the user are transmitted through the car frame to the steering wheel and car seat. These vibrations are then represented by a game controller with a vibrator that replaces the steering wheel, or by a seat-type haptic device. Figure 61 shows the configuration of the haptic data propagation path in such a scene. While such vibration representation can increase the realism of the user's driving operations, it can interfere with operations such as the game's configuration screen. Therefore, for example, when the user is operating such a configuration screen, the use of the steering wheel (Haptic Medium 2A), which is the haptic propagation medium, may be disabled (its state may be turned off), as shown in Figure 61. By doing so, vibrations from the car engine will only be represented by the seat-type haptic device (Haptic Device B). Consequently, interference with the user's input operations to the game controller (Haptic Device A) can be suppressed.

[0252] Furthermore, as shown in Figure 62, if there are multiple haptic sources other than the car engine, such as road bumps or contact with other vehicles, the propagation function of the steering wheel (Haptic Medium 2A) may be similarly turned off to eliminate vibration transmission to the game controller. Alternatively, the haptic sources may be turned off. For example, the monotonous car engine can be turned off so that the user can concentrate on the haptics of road bumps or contact. Moreover, the on / off control of the haptic sources and the on / off control of the haptic data propagation medium may be applied in combination. For example, the car engine may be turned off, and the propagation function of the steering wheel may also be turned off.

[0253] Furthermore, in a haptic listener, vibration propagation through various haptic mediums is mixed. If the level ratio of each mixed haptic data is significantly smaller than one, for example, 100:1, then the computational processing for the propagation of the small haptic data is very inefficient (the impact of omitting it is small). In such cases, the haptic medium constituting that propagation path may be not used (turned off). Alternatively, a priority can be set for the application of haptic mediums, and the application of haptic devices may be controlled according to that priority. For example, suppose the strength of the car engine is "1", the maximum value of the road bump is "1", and the tire suspension attenuates it to 1 / 100. The setting for this tire suspension (Haptic Medium 1B) may be turned off when the processing load of the presentation engine exceeds 80%. Furthermore, processing after the haptic propagation medium is turned off throughout the entire path may be disabled to improve processing efficiency.

[0254] Furthermore, if the car driven by the user comes into contact with an item during the game and the user acquires that item, a haptic medium indicating the acquisition of the item (i.e., a vibration output indicating the acquisition of the item) may be displayed regardless of the point of contact between the item and the car body. In other words, when a predetermined event occurs, a haptic medium (i.e., a vibration output) that is not based on realistic physical relationships or laws may be displayed. To enable such a display, the item may be set as the haptic source, and as a setting value, the attenuation damping function may be selected and set to "noAttenuation" (no damping) as shown in Figure 30B, or the transfer function of the haptic medium that transmits this item haptic source may be set to an equation with no damping. In addition, the use of the desired haptic medium may be turned off, or the propagation of haptic data may be weakened (i.e., the damping rate may be increased).

[0255] Figure 63A shows the arrangement of haptic sources and listeners, as well as the relationship between listeners and haptic devices, in a scene from a car racing video game. Figure 63B shows an example of the propagation path configuration. In Figure 63A, "SA" indicates the source, "contact with another car (Haptic Source A)," "SB" indicates the source, "car engine (Haptic Source B)," "SC" indicates the source, "road bump (Haptic Source C)," and "SD" indicates the source, "item acquisition (Haptic Source D)." "LA" indicates the listener, "steering wheel surface (Haptic Listener A)," and "LB" indicates the listener, "car seat surface (Haptic Listener B)." "DA" indicates the haptic device, "game controller (Haptic Device A)," and "DB" indicates the haptic device, "seat-type haptic device (Haptic Device B)."

[0256] For example, as shown in Figure 63A, when a car driven by a user comes into contact with an item and the user acquires that item, a haptic media indicating the acquisition of the item (i.e., a vibration output indicating item acquisition) may be displayed regardless of the physical relationship with the contact (e.g., the location of contact with the item). In this case, for example, as shown in Figure 63B, the steering wheel (Haptic Medium 2A) may be turned off (i.e., output is not generated from the game controller), and only the sheet-type haptic device may be used to display the haptic media. Furthermore, the parameter "Priority," described later, may be used as a mechanism to prioritize the display of items acquired.

[0257] Furthermore, such propagation path control information may be transmitted. For example, information like that shown in Figure 64 may be transmitted as this propagation path control information. In other words, information like that shown in Figure 64 may be included in the transmitted propagation path control information. For example, this propagation path control information may be transmitted from the encoding device to the decoding device. That is, the encoding device may generate this propagation path control information and transmit it to the decoding device, and the decoding device may use this propagation path control information to derive receptor haptic data.

[0258] For example, in this propagation path control information, a control flag "control" related to the UI configuration setting process may be set. This "control" is flag information that sets whether to turn on or off (use or not) the source, haptic propagation medium, or listener. The user can use this function to control, for example, whether to execute processing for an arbitrary path in the UI configuration menu.

[0259] For example, as shown in Figure 64A, the parameters "index" and "Source control" may be transmitted as propagation path control information (NodeSource) related to the source. This "index" is information that indicates the item referenced in the array of source groups (hapticSources) defined at the glTF file level by the MPEG_spatial_haptic extension. "index" is a required parameter, its data type is "integer", and its initial value is "Not applicable (N / A)". "Source control" specifies that information regarding source control (source control), such as turning the source on / off (muting), should be displayed. This "Source control" is not required, its data type is "boolean", and its initial value is false.

[0260] Furthermore, as shown in Figure 64B, the parameters "index" and "Listener control" may be transmitted as propagation path control information (NodeListener) related to the receptor. This "index" is information that indicates the item referenced in the array of receptor groups (hapticListeners) defined at the glTF file level by the MPEG_spatial_haptic extension. "index" is a required parameter, its data type is "integer", and its initial value is "not applicable (N / A)". "Listener control" specifies that information regarding receptor control (listener control), such as turning the receptor on / off (muting), should be displayed. This "Listener control" is not required, its data type is "boolean", and its initial value is false.

[0261] Furthermore, as shown in Figure 64C, the parameters "index" and "Medium control" may be transmitted as propagation path control information (NodeMedium) related to the haptic propagation medium. This "index" is information that indicates the item referenced in the array of haptic propagation mediums (hapticMediums) defined at the glTF file level by the MPEG_spatial_haptic extension. "index" is a required parameter, its data type is "integer", and its initial value is "Not applicable (N / A)". "Medium control" specifies that information regarding the control of the haptic propagation medium (medium control), such as turning the haptic propagation medium on / off (muting), should be displayed. This "Medium control" is not required, its data type is "boolean", and its initial value is false.

[0262] Furthermore, in the propagation path control information, a default value may be set for the control flag "control" related to the UI configuration setting process.

[0263] For example, as shown in Figure 65A, the parameter "Default control status" may also be transmitted as propagation path control information (NodeSource) related to the source. This "Default control status" indicates the initial value (on or off) of the parameter "Source control". Its data type is "enumeration", and its initial value is "on".

[0264] Similarly, as shown in Figure 65B, the parameter "Default control status" may also be transmitted as propagation path control information (NodeListener) related to the receptor. This "Default control status" indicates the initial value (on or off) of the parameter "Listener control". Its data type is "enumeration", and its initial value is "on".

[0265] Similarly, as shown in Figure 65C, the parameter "Default control status" may also be transmitted as propagation path control information (NodeMedium) related to the haptic propagation medium. This "Default control status" indicates the initial value (on or off) of the parameter "Medium control". Its data type is "enumeration", and its initial value is "on".

[0266] Furthermore, in the propagation path control information, a parameter called "Priority" may be set to specify the process to be executed with the highest priority when simultaneous processing occurs. Users can use this function to reflect the intentions of content creators, for example, by highlighting specific haptics FB.

[0267] For example, as shown in Figure 66A, the parameter "Priority" may be transmitted as propagation path control information (NodeSource) related to the source. This "Priority" defines the priority when a content creator renders multiple sources to the same list simultaneously. The source with the highest priority is processed. Sources with lower priority are not processed. If the priorities are the same, the application must apply its own criteria. The data type for this parameter is "Integer".

[0268] Similarly, as shown in Figure 66B, the parameter "Priority" may be transmitted as propagation path control information (NodeListener) for the receptor. This "Priority" defines the priority when a content creator renders the same source simultaneously on different receptors. The receptor with the highest priority is processed. Receptors with lower priority are not processed. If the priorities are the same, the application must apply its own criteria. The data type for this parameter is "Integer".

[0269] Similarly, as shown in Figure 66C, the parameter "Priority" may be transmitted as propagation path control information (NodeMedium) for the haptic propagation medium. This "Priority" defines the priority when a content creator processes multiple haptic propagation media simultaneously. The haptic propagation medium with the highest priority is processed. Haptic propagation media with a lower priority are not processed. If the priorities are the same, the application must apply its own criteria. The data type for this parameter is "Integer".

[0270] Similarly, for the haptic pipeline (MPEG_spatial_haptics.Pipeline) shown in Figure 56, in addition to the parameter "pipelineInformation" in Figure 57, the parameter "Priority" may also be transmitted as control information for grouped propagation paths. This "Priority" defines the priority when a content creator processes multiple grouped propagation paths simultaneously. The grouped propagation path with the highest priority is processed. Grouped propagation paths with lower priority are not processed. If the priorities are the same, the application must apply its own criteria. The data type for this parameter is "Integer".

[0271] Furthermore, in this propagation path control information, a parameter called "Priority" may be set to prioritize processing according to the terminal's processing capacity when simultaneous processing occurs. Users can use this function to reflect the intentions of content creators, for example, by processing the appropriate haptics FB with an appropriate amount of processing power.

[0272] For example, as shown in Figure 67A, the parameter "Priority" may be transmitted as propagation path control information (NodeSource) related to the source. This "Priority" defines the priority when a content creator renders multiple sources simultaneously to the same list. Sources with higher priority are processed. Sources with lower priority are processed. If the priorities are the same, the application must apply its own criteria. The data type for this parameter is "Integer".

[0273] Similarly, as shown in Figure 67B, the parameter "Priority" may be transmitted as propagation path control information (NodeListener) for the receptor. This "Priority" defines the priority when content creators render the same source simultaneously on different receptors. Receptors with higher priority are processed. Receptors with lower priority are not processed. If the priority is the same, the application must apply its own criteria. The data type for this parameter is "Integer".

[0274] Similarly, as shown in Figure 67C, the parameter "Priority" may be transmitted as propagation path control information (NodeMedium) for the haptic propagation medium. This "Priority" defines the priority when a content creator processes multiple haptic propagation media simultaneously. Haptic propagation media with a higher priority are processed. Haptic propagation media with a lower priority are not processed. If the priorities are the same, the application must apply its own criteria. The data type for this parameter is "Integer".

[0275] Similarly, for the haptic pipeline (MPEG_spatial_haptics.Pipeline) shown in Figure 56, in addition to the parameter "pipelineInformation" in Figure 57, the parameter "Priority" may also be transmitted as control information for grouped propagation paths. This "Priority" defines the priority when a content creator processes multiple grouped propagation paths simultaneously. Grouped propagation paths with higher priority are processed. Grouped propagation paths with lower priority are not processed. If the priority is the same, the application must apply its own criteria. The data type for this parameter is "Integer".

[0276] Furthermore, target sources, haptic propagation media, and receptors may be set for the propagation path control information. For example, when simultaneous processing occurs, a specific processing path including that target may be selected (or prioritized). Users can use this function to reflect the intent of content creators, such as highlighting specific haptics FBs.

[0277] For example, as shown in Figure 68A, the parameter "Target listener" may be transmitted as propagation path control information (NodeSource) related to the source. This "Target listener" is information that specifies the receptor (hereinafter also referred to as the target receptor) that renders the haptic data propagated from this source. This target receptor is set based on the intent of the content creator. If this "Target listener" is set, the haptic data propagated from this source will not be processed by receptors other than the target receptor. The data type is "Array". In other words, the target receptor is specified by an array of receptor indices.

[0278] Similarly, as shown in Figure 68B, the parameter "Target source" may be transmitted as propagation path control information (NodeListener) for the receptor. This "Target source" specifies the source of the haptic data to be rendered by this receptor (hereinafter also referred to as the target source). This target source is set based on the intent of the content creator. If this "Target source" is set, this receptor will not process haptic data propagated from sources other than the target source. The data type is "Array". In other words, the target source is specified by an array of source indices.

[0279] Similarly, as shown in Figure 68C, the parameters "Target listener" and "Target source" may be transmitted as propagation path control information (NodeMedium) related to the haptics propagation medium. This "Target listener" specifies the target receptor that renders the haptics data propagated through this haptics propagation medium. This target receptor is set based on the content creator's intent. If this "Target listener" is set, the haptics data propagated through this haptics propagation medium will not be processed by receptors other than the target receptor. The data type is "Array". In other words, the target receptor is specified by an array of receptor indices.

[0280] Furthermore, "Target source" is information that specifies the target source of haptic data propagated through this haptic propagation medium. This target source is set based on the content creator's intent. If this "Target source" is set, this haptic propagation medium will not process haptic data propagated from sources other than the target source. The data type is "Array". In other words, the target source is specified by an array of source indices.

[0281] Furthermore, as shown in Figure 69, the parameter "Process control threshold" may be transmitted as propagation path control information (NodeListener) related to the receptor. This "Process control threshold" is a parameter that defines a threshold (%) for controlling the process (execution of processing). This "Process control threshold" is set based on the intent of the content creator. For example, when rendering haptic data from multiple sources, the receptor is controlled not to render haptic data whose level is below the threshold (ratio) set as this "Process control threshold" compared to the maximum level of haptic data. Alternatively, the processing of the propagation path of that haptic data may be prevented (for example, the haptic propagation medium constituting that propagation path may be turned off). In this way, for example, processing efficiency can be increased without compromising the user experience, within the scope of the content creator's intent.

[0282] <Flow of Regeneration Process> This technology (Method 1-11) can be applied to any device. For example, this technology (Method 1-11) may be applied to the regeneration device 300 shown in Figure 34. In that case, the regeneration device 300 has the same configuration as the example in Figure 34. An example of the flow of the regeneration process performed by the regeneration device 300 in this case will be explained with reference to the flowchart in Figure 70.

[0283] In this case, in step S721, the haptics decoding unit 311 decodes the bitstream of source haptics data. In step S722, the haptics propagation path length derivation unit 321 determines the source location and receptor location, and derives the haptics propagation path length based on these and the propagation path configuration information. In step S723, the haptics propagation characteristic setting unit 323 sets the haptics propagation characteristics based on the source location, receptor location, and propagation path configuration information. In step S724, the receptor haptics data generation unit 322 generates receptor haptics data using the source haptics data, the haptics propagation path length, the haptics propagation characteristics, and the propagation path control information described above. In step S725, the haptics presentation unit 313 outputs the receptor haptics data.

[0284] By executing each process in this manner, a wider variety of haptic experiences can be provided. Furthermore, an increase in processing load can be suppressed.

[0285] Furthermore, the propagation path control information may be transmitted from the encoding device to the decoding device. In other words, this technology (Method 1-11) may be applied to the file generation device 500 shown in Figure 44. In that case, the file generation device 500 may store the propagation path control information as haptics-related information in the scene description. Therefore, in this case as well, the file generation device 500 has the same configuration as the example in Figure 44. The file generation device 500 then executes the file generation process in the same flow as the flowchart in Figure 45. In other words, this technology (Method 1-11) may be applied to the file generation process shown in Figure 45.

[0286] Furthermore, this technology (Method 1-11) may be applied to the playback device 600 shown in Figure 46. In that case, the playback device 600 may acquire a scene description including haptics-related information including propagation path control information, and perform processing related to the generation of receptor haptics data based on that scene description (i.e., propagation path control information). Therefore, in this case as well, the playback device 600 has the same configuration as in Figure 46. The playback device 600 then performs the playback process in the same flow as the flowchart in Figure 47. In other words, this technology (Method 1-11) may be applied to the playback process shown in Figure 47.

[0287] <Method 1-12> For example, during the game, the avatar may switch to a predetermined mode such as "astral body" mode or undergo a transformation, which may cause the location and number of receptors to change or disappear. For example, the sensitivity of the avatar's entire body's receptors in "astral body" mode may be set to 30% of that in normal mode to reduce the impact of object contact. In this way, "contact in another world" (contact with a different sensation than in normal mode) can be expressed in "astral body" mode.

[0288] Furthermore, in modes such as "Focus" mode, a specific part of the avatar may be set as a region of focus, and only the receptors in that region of focus may be enabled (other receptors may be disabled). For example, in "Focus on Hands" mode, the sensitivity of receptors other than those in the avatar's "hands" may be set to 30% of the normal mode, and the sensitivity of the receptors in the avatar's "hands" may be set to 150% of the normal mode to represent a state where only the "hands" are sensitive. In other words, the haptic expression of the avatar's "hands" may be emphasized. In addition, in certain modes such as "Focus" mode, unique haptic effects may be performed in combination with visuals, etc. For example, to emphasize the moment of impact of a batted ball in baseball, haptic expression may be performed in combination with visual effects.

[0289] Furthermore, in modes such as "anesthesia" or "invisibility," effects that alter the avatar's body sensations may be applied. Haptics may also be used as feedback in open-hand devices for Augmented Reality (AR) and Mixed Reality (MR). For example, in a battle game, a status such as "paralysis" could be represented by turning off (or setting the sensitivity to 0) the receptors in the paralyzed area, thereby representing a situation where there is no haptic feedback.

[0290] As various haptics can be presented in this way, there was a risk that the haptics that content creators wanted to draw attention to might get lost in the mix. For example, there was a demand to arbitrarily select the receptors that receive the haptic data that the user wanted to draw attention to within a scene. There was also a demand to present haptics according to the state of the avatar (for example, modes such as "focus on hands" or "paralysis"). However, conventional haptic presentation methods could not adequately meet these demands. Therefore, when Method 1 is applied, the use of receptors (Listeners) may be controlled as shown in the third row from the top of the table in Figure 48 (Method 1-12).

[0291] For example, in a decoding device, the receptor haptics data generation unit may further generate receptor haptics data sensed by the receptor using receptor control information that controls the use of the receptor.

[0292] For example, this receptor control information may indicate whether or not to use a receptor, and the receptor haptics data generation unit may generate receptor haptics data sensed by the receptor indicated by the receptor control information to be used.

[0293] Alternatively, this receptor control information may indicate the sensitivity of the receptor, and the receptor haptics data generation unit may use the sensitivity indicated by the receptor control information to generate receptor haptics data that the receptor senses.

[0294] In other words, receptors may be equipped with on / off functions and sensitivity adjustment functions so that users can select and adjust the receptors they want to focus on. This makes it easier to achieve the intended expression of the content creator. It also allows users' preferences to be reflected across different devices. For example, the same content can be experienced in the same way on devices with various specifications.

[0295] Furthermore, such receptor control information may be transmitted. For example, the information shown in Figure 71 may be transmitted as this receptor control information. In other words, the information shown in Figure 71 may be included in the transmitted receptor control information. For example, this receptor control information may be transmitted from the encoding device to the decoding device. That is, the encoding device may generate this receptor control information and transmit it to the decoding device, and the decoding device may use this receptor control information to derive receptor haptics data.

[0296] For example, as shown in Figure 71A, the parameter "index" may be transmitted as receptor control information (NodeListener). This "index" is information that indicates the item referenced in the sequence of receptor groups (hapticListeners) defined at the glTF file level by the MPEG_spatial_haptic extension. "index" is a required parameter, its data type is "integer", and its initial value is "Not applicable (N / A)".

[0297] Furthermore, as shown in Figure 71B, a list of haptic listener status may be transmitted as receptor control information. This list may include the parameters "HapticListenerStatusName" and "HapticListenerStatus". "HapticListenerStatusName" is information indicating the name of the receptor status. Its data type is "string". "HapticListenerStatus" is information that sets the receptor index and its sensitivity for each receptor status. The sensitivity may be set, for example, in "%". Normally, it is set in the range of 0% to 100%, but if emphasis is needed, it may be possible to set a value greater than 100%. Its data type is "Array". In other words, "HapticListenerStatus" is set as an array of receptor indices and their sensitivities.

[0298] <Flow of Regeneration Process> This technology (Method 1-12) can be applied to any device. For example, this technology (Method 1-12) may be applied to the regeneration device 300 shown in Figure 34. In that case, the regeneration device 300 has the same configuration as the example in Figure 34. An example of the flow of the regeneration process performed by the regeneration device 300 in this case will be explained with reference to the flowchart in Figure 72.

[0299] In this case, in step S741, the haptics decoding unit 311 decodes the bitstream of source haptics data. In step S742, the haptics propagation path length derivation unit 321 determines the source location and receptor location, and derives the haptics propagation path length based on these and the propagation path configuration information. In step S743, the haptics propagation characteristic setting unit 323 sets the haptics propagation characteristics based on the source location, receptor location, and propagation path configuration information. In step S744, the receptor haptics data generation unit 322 generates receptor haptics data using the source haptics data, the haptics propagation path length, the haptics propagation characteristics, and the receptor control information described above. In step S745, the haptics presentation unit 313 outputs the receptor haptics data.

[0300] By performing each process in this manner, it becomes easy to express content exactly as the creator intended and to reflect user preferences across different devices.

[0301] Furthermore, the receptor control information may be transmitted from the encoding device to the decoding device. In other words, this technology (Method 1-12) may be applied to the file generation device 500 shown in Figure 44. In that case, the file generation device 500 may store the receptor control information as haptics-related information in the scene description. Therefore, in this case as well, the file generation device 500 has the same configuration as the example in Figure 44. The file generation device 500 then performs the file generation process in the same flow as the flowchart in Figure 45. In other words, this technology (Method 1-12) may be applied to the file generation process shown in Figure 45.

[0302] Furthermore, this technology (Method 1-12) may be applied to the playback device 600 shown in Figure 46. In that case, the playback device 600 may acquire a scene description including haptics-related information including receptor control information, and perform processing related to the generation of receptor haptics data based on that scene description (i.e., receptor control information). Therefore, in this case as well, the playback device 600 has the same configuration as in Figure 46. The playback device 600 then performs the playback process in the same flow as the flowchart in Figure 47. In other words, this technology (Method 1-12) may be applied to the playback process shown in Figure 47.

[0303] <Method 1-13> By having a haptic device represent how receptors sense haptic data, users can obtain a haptic experience as a real-world experience. In other words, the haptic device (or its actuators) needs to be associated with some kind of receptor. However, the specifications of haptic devices are becoming increasingly diverse. For example, the shape and attachment site of the haptic device, the type, position, and number of actuators are likely to differ from device to device. Also, many new devices are being developed.

[0304] If the correspondence between receptors and haptic data (and its actuators) breaks down (i.e., a discrepancy arises between the pre-configured settings and the actual correspondence), the haptic device may not be able to produce appropriate vibration output, potentially making it difficult for the user to obtain a proper haptic experience. For example, if haptic data sensed by a receptor is represented by haptic device A and haptic device B, which have different specifications, the representations may differ. In other words, at least one of the devices may not be able to provide the user with a proper haptic experience.

[0305] Thus, even if haptic data is created assuming the use of a specific haptic device, there is a possibility that a haptic device with different specifications may actually be applied, potentially making it difficult for users to obtain an appropriate haptic experience. In other words, it was highly likely that establishing a fixed correspondence between receptors and haptic data in advance would be practically impossible.

[0306] Therefore, the haptic device may be made capable of establishing a correspondence between receptors and the haptic device itself. For example, the correspondence between receptors and haptic devices (actuators) may be arbitrarily set, such as one-to-many or many-to-one. For example, receptors assigned to the avatar may be efficiently distributed to the actuators of a real-world haptic device, taking into account standard positions and optimal resolution and sensitivity based on human characteristics. Also, since simply synthesizing various haptic data may result in noise, it may be possible to control the on / off state of specific areas or ranges.

[0307] For example, when Method 1 is applied, the receptor (Listener) and the haptic device may be linked as shown in the fourth row from the top of the table in Figure 48 (Method 1-13). For example, the decoding device may further include a haptic display unit that outputs physical motion corresponding to receptor haptic data sensed by the receptor linked to it. For example, the haptic display unit may output physical motion corresponding to synthesized receptor haptic data obtained by synthesizing the receptor haptic data sensed by each of the multiple receptors linked to it.

[0308] For example, as shown in Figure 73A, multiple receptors (L1 to L3) and multiple haptic device actuators (D1 to D3) may be associated with each other on a one-to-one basis. For example, receptor L1 may be associated with actuator D1, receptor L2 with actuator D2, and receptor L3 with actuator D3 (one-to-one association).

[0309] Furthermore, as shown in Figure 73B, multiple receptors (L1 to L3) and multiple haptic device actuators (D1 to D3) may be arranged in a one-to-one correspondence by rearranging (displacement) their corresponding order. For example, receptor L1 may be associated with actuator D3, receptor L2 with actuator D1, and receptor L3 with actuator D2 (displacement).

[0310] Furthermore, as shown in Figure 73C, some of the multiple receptors (L1 to L3) may be selected and associated with a single haptic device actuator (D1). For example, receptor L1 may be selected from receptors L1, L2, and L3 and associated with actuator D1 (selection and extraction).

[0311] Furthermore, as shown in Figure 73D, multiple receptors (L1 to L3) may be associated with a single haptic device actuator (D1). For example, receptors L1, L2, and L3 may all be associated with actuator D1 (MIX). In this case, the haptic data sensed by each receptor is combined, and vibrations corresponding to the combined haptic data are output from actuator D1.

[0312] Furthermore, as shown in Figure 74A, multiple receptors (L1 to L3) may be weighted and associated with a single haptic device actuator (D1). For example, receptors L1, L2, and L3 may each be weighted and associated with actuator D1 (weighting). In this case, the haptic data sensed by each receptor is weighted and synthesized, and vibrations corresponding to the synthesized haptic data are output from actuator D1. The weight values ​​assigned to each receptor may be fixed or variable. For example, the weight values ​​may change depending on the scene or case. For example, the weight values ​​may be set according to whether multiple haptic data exist simultaneously, whether or not haptic data exists within an activation period within a certain time, amplitude, location, ergonomics, etc.

[0313] Furthermore, as shown in Figure 74B, multiple receptors (L1 to L3) may be weighted and associated with multiple haptic device actuators (D1, D2). For example, receptors L1, L2, and L3 may be weighted and associated with actuators D1 and D2 respectively (weighted integration). In this case, the haptic data sensed by each receptor is weighted and synthesized, and vibrations corresponding to this synthesized haptic data are output from actuators D1 and D2. Note that the weight values ​​of each receptor may be changed for each actuator.

[0314] Furthermore, as shown in Figure 74C, a single receptor (L1) may be associated with a portion of the actuators (D1 to D3) of multiple haptic devices. For example, receptor L1 may be associated with actuator D1 selected from actuators D1 to D3 (optimal selection / decimation). For example, in actuator selection, the actuator best suited to receptor L1 (or the haptic data sensed by receptor L1) may be selected. Also, in actuator selection, multiple actuators may be selected. For example, receptor L1 may be associated with selected actuators D1 and D2.

[0315] Furthermore, as shown in Figure 74, D, a single receptor (L1) may be associated with multiple haptic device actuators (D1 to D3). For example, receptor L1 may be associated with each of actuators D1 to D3 (distribution / total presentation). In this case, vibrations corresponding to the haptic data sensed by receptor L1 may be output from each actuator, or the vibrations corresponding to the haptic data sensed by receptor L1 may be divided into three equal parts, and these divided vibrations may be output from each actuator.

[0316] Furthermore, as shown in Figure 74E, multiple receptors (L1 and L2) may be weighted and associated with multiple haptic device actuators (D1 to D3). For example, receptor L1 may be weighted and associated with actuators D1 to D3. Similarly, receptor L2 may be weighted and associated with actuators D1 to D3 (weighted distribution). In this case, the vibrations corresponding to the haptic data sensed by receptor L1 or receptor L2 may be divided into three parts according to the weight values ​​of each actuator, and these divided vibrations may be output from each actuator. The weight values ​​assigned to each haptic device (actuator) may be fixed or variable. For example, the weight values ​​may change depending on the scene or case. For example, the weight values ​​may be set according to whether multiple haptic data exist simultaneously, whether or not haptic data exists within an activation time within a certain period, amplitude, location, ergonomics, etc.

[0317] By establishing this mapping across haptic devices, users can obtain a haptic experience appropriate to the scene, regardless of the specific haptic device being used.

[0318] Furthermore, receptor-device linking information that associates such receptors with haptic devices (actuators) may be transmitted. For example, the information shown in Figure 75 may be transmitted as this receptor-device linking information. In other words, the information shown in Figure 75 may be included in the transmitted receptor-device linking information. For example, this receptor-device linking information may be transmitted from the encoding device to the decoding device. That is, the encoding device may generate this receptor-device linking information and transmit it to the decoding device, and the decoding device may use this receptor-device linking information to output vibrations corresponding to haptic data.

[0319] For example, as shown in Figure 75A, the parameter "index" may be transmitted as receptor device association information (Node Listener) related to the receptor. This "index" is information that indicates the item referenced in the sequence of receptor groups (hapticListeners) defined at the glTF file level by the MPEG_spatial_haptic extension. "index" is a required parameter, its data type is "integer", and its initial value is "Not applicable (N / A)".

[0320] Furthermore, as shown in Figure 75B, the parameter "index" may be transmitted as receptor device association information (Haptic Device Actuator) related to the actuator of the haptic device. This "index" is information indicating the item referenced in the array of actuators of the haptic device. Its data type is "integer," and its initial value is "Not applicable (N / A)."

[0321] Furthermore, as shown in Figure 75C, the parameter "mapping" may be transmitted as receptor device linking information. This "mapping" is information that shows the relationship between the receptor and the actuator as a matrix, and its data type is "Array".

[0322] By transmitting this kind of information, the linking of receptors and haptic devices (actuators) can be done according to the content creator's intentions.

[0323] This mapping between receptors and haptic devices (actuators) can be applied in various situations. For example, the specifications of a haptic device may change due to version upgrades. For instance, as shown in Figure 76A, the number of actuators (gray circles in the figure) in a sheet-type haptic device may increase with each version upgrade. Similarly, as shown in Figure 76B, the number of actuators (gray circles in the figure) in a clothing-type device may increase with each version upgrade. As described above, by mapping receptors to haptic devices (actuators), such specification changes can be reflected in the correspondence between receptors and haptic devices (actuators). Therefore, even if the specifications of a haptic device change due to a version upgrade, users can still obtain a haptic experience appropriate to the scene.

[0324] Furthermore, situations involving haptic experiences for users are diverse. For example, games and applications that involve haptic experiences for users come in a variety of specifications, such as third-person games where avatars move around in various fields, racing games with feedback from the seat and steering wheel regarding engine, road conditions, and collisions with other vehicles, avatar interaction games where users can handle and touch various weapons, and first-person games. In this way, haptic devices may be used for diverse games and applications where the settings of receptors and haptic data differ from game to game. In such cases, the haptic experience expressed by the haptic device will differ from game to game or application to application.

[0325] In some cases, it may be necessary to represent multiple haptic experiences with a single haptic device. For example, as shown in Figure 77A, when using a smartphone or game controller with a vibrator, it may be possible to present haptic experiences other than touching with the hands, such as full-body damage or the feeling of stepping on terrain, using these devices. Also, as shown in Figure 77B, when using a game controller or a sheet-type haptic device, it may be possible to present haptic experiences such as damage and sensations received on parts other than the hands, the perception of vibrations in the space, and feedback received on the hands when touching objects or using tools. Furthermore, as shown in Figure 77C, when using a glove-type or clothing-type haptic device, it may be possible to present haptic experiences such as damage and sensations received on parts other than the hands, the perception of vibrations in the space, and more subtle feedback received on the hands when touching objects or using tools, using these devices. As described above, by establishing a correspondence between receptors and the haptic device (its actuators), it is possible to respond to various situations like these, allowing users to obtain a haptic experience appropriate to the scene.

[0326] Furthermore, depending on the situation, it may be necessary to change the receptors corresponding to the actuators of the haptic device. For example, in a baseball game, as shown in Figure 78A, the assignment of the haptic device's actuators to either the left or right hand of the avatar may change depending on whether the batter is right-handed or left-handed. As described above, by mapping receptors to the haptic device (its actuators), such changes can be made, allowing the user to obtain a haptic experience appropriate to the scene. The body part associated with an actuator may be fixed or variable. It may also be able to handle several body parts, or the assigned body part may change depending on the scene.

[0327] Furthermore, as shown in Figure 78B, the receptors and haptic devices (actuators) may be associated with each other, taking into account human characteristics. For example, the resolution may be finer for fingers and coarser for the back, taking into account human discrimination thresholds. Alternatively, the resolution may be artificially increased by weighting and distributing information to intermediate positions using phantom sensations or similar methods. As described above, by associating receptors with haptic devices (actuators), such diverse expressions become possible, allowing users to obtain haptic experiences appropriate to the scene.

[0328] <Mapping Receptors to Virtual Points of Action with Devices> For example, receptors mapped to points of action can be placed in a virtual space, allowing them to be embodied as an extended body. Intuitive haptic feedback can also be obtained with open-hand devices.

[0329] Traditional methods allow for the representation of an augmented body in virtual space, but there was no established method for presenting these haptic experiences to the real body. For example, when using open-hand devices, it was not intuitively clear what kind of haptic experience contact in virtual space would produce for the device.

[0330] Therefore, the correspondence between the receptors of the virtual avatar and the haptic device (actuator) may be made intuitive. For example, in Figure 79A, the gray area represents the user's hand in the real world. The dotted line represents the part of the avatar that does not correspond to the user's body. "L" represents a receptor, and "D" represents a haptic device (actuator). As shown in Figure 79A, a haptic listener L may be placed as a point of action in the virtual world on the part of the avatar that does not correspond to the user's body. Alternatively, a haptic device (actuator) D may be placed on the user's finger in the real world. The haptic experience perceived by the receptor L in the virtual world may then be represented by the haptic device D in the real world. In other words, in this case, the haptic experience perceived in the extended body (the part of the avatar that does not correspond to the user's real body) can be provided to the user via the haptic device D.

[0331] Alternatively, the receptor L in the virtual space may be placed on the part of the avatar that corresponds to the user's real body. For example, as shown in Figure 79B, the receptor L may be placed in approximately the same position as the haptics device D. In this way, the receptor L senses the haptic experience received, for example, at the tip of the avatar's hand, as it propagates through the avatar. In other words, the haptics device D represents the haptic experience as it propagates through the extended body. Therefore, the haptic experience received in the virtual space can be represented more faithfully in the real world.

[0332] Alternatively, as shown in Figure 79C, for example, a receptor L may be placed at the tip of the avatar's hand, and a haptics device (actuator) D may be placed at the user's real wrist. For example, the haptics experience received in the avatar may be presented at another location on the body, such as an open-hand device.

[0333] Alternatively, for example, as shown in Figure 79D, a receptor L may be placed on the avatar's wrist, and a haptic device (actuator) D may be placed on the user's real wrist. Then, an intuitively understandable link may be established (visually connected, displaying mechanical links or signal lines, etc.), and presented to the user as a haptic experience through that link. In this way, the haptic device D can represent a haptic experience propagating through an extended body. Therefore, the haptic experience received in virtual space can be more faithfully represented in the real world.

[0334] Alternatively, as shown in Figure 79E, for example, receptors (L1 to L5) may be placed at the tips of each finger, and a haptic device D may be placed on the wrist, and the haptic experience sensed by each receptor L may be represented on the haptic device D. For example, an intuitively understandable link may be set up (visually connected, displaying mechanical links or signal lines, etc.), and the haptic experience may be presented to the user through that link. In this way, the haptic device D can represent the haptic experience sensed by each receptor L.

[0335] By doing so, it becomes easier to establish intuitive links that are easily understood physically and empirically, enabling an intuitive grasp of haptic experiences. Furthermore, various devices can operate on the same platform. It can also be used as a user interface (UI) for robots that use real sensors instead of receptors (Haptic Listeners). For example, haptic devices can be used as open-hand controllers.

[0336] <Flow of Regeneration Process> This technology (Method 1-13) can be applied to any device. For example, this technology (Method 1-13) may be applied to the regeneration device 300 shown in Figure 34. In that case, the regeneration device 300 has the same configuration as the example in Figure 34. An example of the flow of the regeneration process performed by the regeneration device 300 in this case will be explained with reference to the flowchart in Figure 80.

[0337] In this case, in step S761, the haptics decoding unit 311 decodes the bitstream of source haptics data. In step S762, the haptics propagation path length derivation unit 321 determines the source location and the receptor location, and derives the haptics propagation path length based on them. In step S763, the haptics propagation characteristic setting unit 323 determines that the receptor haptics data generation unit 322 generates receptor haptics data using the source haptics data, the haptics propagation path length, and the haptics propagation characteristics. In step S764, the haptics presentation unit 313 outputs receptor haptics data based on the receptor device association information.

[0338] By executing each process in this manner, a more appropriate haptic experience can be provided to the user.

[0339] Furthermore, the receptor device association information may be transmitted from the encoding device to the decoding device. In other words, this technology (Method 1-13) may be applied to the file generation device 500 shown in Figure 44. In that case, the file generation device 500 may store the receptor device association information as haptics-related information in the scene description. Therefore, in this case as well, the file generation device 500 has the same configuration as the example in Figure 44. The file generation device 500 then executes the file generation process in the same flow as the flowchart in Figure 45. In other words, this technology (Method 1-13) may be applied to the file generation process shown in Figure 45.

[0340] Furthermore, this technology (Method 1-13) may be applied to the playback device 600 shown in Figure 46. In that case, the playback device 600 may acquire a scene description including haptics-related information including receptor device association information, and perform processing related to the output of receptor haptics data based on that scene description (i.e., receptor device association information). Therefore, in this case as well, the playback device 600 has the same configuration as in Figure 46. The playback device 600 then performs the playback process in the same flow as the flowchart in Figure 47. In other words, this technology (Method 1-13) may be applied to the playback process shown in Figure 47.

[0341] <Method 1-14> For example, as shown in the graph in Figure 81A, if the haptic propagation characteristics (attenuation function) depend only on distance, then even if the amplification factor differs between frequencies, as shown in the graph in Figure 81B, the attenuation characteristics at each frequency will be similar, as shown in Figure 81C. Therefore, there was a risk that the expressiveness (accuracy) of the haptic propagation characteristics would be reduced.

[0342] Therefore, the haptic propagation characteristics may be represented as a two-dimensional array that depends on both frequency f and distance r. For example, when Method 1 is applied, the haptic propagation characteristics may be represented as a two-dimensional array, as shown in the bottom row of the table in Figure 48 (Method 1-14).

[0343] For example, as shown in Figure 82A, frequency-dependent distance attenuation characteristics may be set. For example, in a decoding device, the haptic propagation characteristics may represent the frequency-dependent propagation characteristics of haptic data by the haptic propagation medium constituting the propagation path.

[0344] If the amplification factor with respect to frequency has the characteristics shown in graph B of Figure 82, then the attenuation characteristics for each frequency will have the characteristics shown in graph C of Figure 82. In other words, the attenuation characteristics can change with each frequency. By doing so, a wider variety of attenuation characteristics can be expressed, thereby suppressing a reduction in the expressiveness (accuracy) of the haptic device characteristics. For example, it becomes possible to express that high frequencies are felt at the root, but the proportion of low-frequency components increases as the distance r increases.

[0345] The frequency domain haptic propagation characteristics between two-dimensional arrays may be interpolated. Alternatively, they may be represented by a three-dimensional graph, as shown in the example in Figure 83.

[0346] This technology (Method 1-14) can be applied to any device. For example, this technology (Method 1-14) may be applied to the regeneration device 300 shown in Figure 34. In that case, the regeneration device 300 has the same configuration as the example in Figure 34 and performs the regeneration process in the same flow as the example in Figure 35, etc. In other words, this technology (Method 1-14) may also be applied to the regeneration process shown in Figure 35, etc.

[0347] Furthermore, the haptic propagation characteristics may be transmitted from the encoding device to the decoding device. In other words, this technique (Method 1-14) may be applied to the file generation device 500 shown in Figure 44. In that case, the file generation device 500 may store its haptic propagation characteristics as haptic-related information in the scene description. Therefore, in this case as well, the file generation device 500 has the same configuration as the example in Figure 44. The file generation device 500 then executes the file generation process in the same flow as the flowchart in Figure 45. In other words, this technique (Method 1-14) may be applied to the file generation process shown in Figure 45.

[0348] Furthermore, this technology (Method 1-14) may be applied to the playback device 600 shown in Figure 46. In that case, the playback device 600 may acquire a scene description including haptics-related information, including haptics propagation characteristics, and perform processing related to the generation of receptor haptics data based on that scene description (i.e., haptics propagation characteristics). Therefore, in this case as well, the playback device 600 has the same configuration as in Figure 46. The playback device 600 then performs the playback process in the same flow as the flowchart in Figure 47. In other words, this technology (Method 1-14) may be applied to the playback process shown in Figure 47.

[0349] <8. Addendum> <Computer> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0350] Figure 84 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0351] In the computer 1900 shown in Figure 84, the CPU (Central Processing Unit) 1901, ROM (Read Only Memory) 1902, and RAM (Random Access Memory) 1903 are interconnected via a bus 1904.

[0352] An input / output interface 1910 is also connected to the bus 1904. The input / output interface 1910 is connected to an input unit 1911, an output unit 1912, a storage unit 1913, a communication unit 1914, and a drive 1915.

[0353] The input unit 1911 consists of, for example, a keyboard, mouse, microphone, touch panel, and input terminals. The output unit 1912 consists of, for example, a display, speaker, and output terminals. The storage unit 1913 consists of, for example, a hard disk, RAM disk, and non-volatile memory. The communication unit 1914 consists of, for example, a network interface. The drive 915 drives removable media 1921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0354] In a computer configured as described above, the CPU 1901 loads, for example, a program stored in the memory unit 1913 into the RAM 1903 via the input / output interface 1910 and the bus 1904, and executes it, thereby performing the series of processes described above. The RAM 1903 also appropriately stores data necessary for the CPU 1901 to perform various processes.

[0355] The program to be executed by the computer can be recorded and applied on a removable medium 1921, such as a package medium. In this case, the program can be installed in the storage unit 1913 via the input / output interface 1910 by inserting the removable medium 1921 into the drive 1915.

[0356] Furthermore, this program can also be provided via wired or wireless transmission media such as local area networks, the internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 1914 and installed in the storage unit 1913.

[0357] Additionally, this program can be pre-installed in ROM 1902 or memory unit 1913.

[0358] <Applications of this technology> This technology can be applied to any configuration. For example, this technology can be applied to various electronic devices.

[0359] Furthermore, this technology can also be implemented as part of a device, such as a processor as a system LSI (Large Scale Integration) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set).

[0360] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0361] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0362] <Applicable Fields and Applications of This Technology> Systems, devices, and processing units incorporating this technology can be used in any field, such as transportation, medicine, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, the applications are entirely arbitrary.

[0363] <Other> In this specification, "flag" refers to information used to identify multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the values ​​that this "flag" can take are, for example, two values, 1 / 0, or three or more values. In other words, the number of bits that constitute this "flag" is arbitrary, and can be 1 bit or multiple bits. Furthermore, identification information (including flags) is envisioned not only in the form of including the identification information itself in the bitstream, but also in the form of including difference information of the identification information relative to a certain reference information in the bitstream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information relative to the reference information.

[0364] Furthermore, various types of information (metadata, etc.) related to encoded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the encoded data. Here, the term "associate" means, for example, making it possible to use (link) one data when processing the other. In other words, associated data may be combined into a single data, or they may be individual data. For example, information associated with encoded data (image) may be transmitted on a different transmission path than the encoded data (image). Also, for example, information associated with encoded data (image) may be recorded on a different recording medium (or a different recording area on the same recording medium) than the encoded data (image). Note that this "association" may not apply to the entire data, but only to a part of it. For example, an image and the information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a part within a frame.

[0365] In this specification, terms such as "combine," "multiplex," "add," "integrate," "include," "store," "insert," "insert," and "place" mean combining multiple things into one, such as combining encoded data and metadata into a single data, and represent one method of "associating" as described above.

[0366] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0367] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0368] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0369] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0370] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0371] Furthermore, for example, multiple technologies relating to this technology can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be implemented in combination with some or all of the technologies described in another embodiment. Also, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0372] Furthermore, this technology can also be configured as follows: (1) A decoding device comprising: a decoding unit that decodes a bitstream to generate source haptic data, which is haptic data at the source; and a receptor haptic data generation unit that generates receptor haptic data, which is haptic data sensed by the receptor, using the source haptic data, the haptic propagation path length, which is the path length of the propagation path of the source haptic data from the source to the receptor, and haptic propagation characteristics that indicate the propagation characteristics of the haptic data due to the haptic propagation medium constituting the propagation path. (2) The decoding device according to (1), wherein the haptic propagation medium is space, and the receptor haptic data generation unit generates the receptor haptic data using the source haptic data, an arbitrary haptic propagation path length in the space, and the haptic propagation characteristics due to the components or state of the space. (3) The decoding device according to (1) or (2), wherein the haptic propagation medium is an object, and the receptor haptic data generation unit generates the receptor haptic data using the source haptic data, the haptic propagation path length according to the shape of the object, and the haptic propagation characteristics according to the material, shape, or state of the object. (4) The decoding device according to any one of (1) to (3), further comprising a haptic propagation characteristic setting unit for setting the haptic propagation characteristics, wherein the receptor haptic data generation unit generates the receptor haptic data using the set haptic propagation characteristics. (5) The decoding device according to (4), wherein the haptic propagation characteristic setting unit selects a function to be used as the haptic propagation characteristics from among a plurality of candidates. (6) The decoding device according to (4) or (5), wherein the haptic propagation characteristic setting unit sets the coefficients of a predetermined model of the haptic propagation characteristics. (7) The decoding device according to any one of (1) to (6), further comprising a detection unit for detecting predetermined information relating to the external environment, and a haptic propagation characteristic setting unit for setting the haptic propagation characteristics based on the detected values.(8) The decoding device according to (7), wherein the detection unit detects information relating to the contact state between an object and a user in a scene, and the haptics propagation characteristic setting unit sets the haptics propagation characteristics based on the contact state indicated by the detected value. (9) The decoding device according to (7) or (8), wherein the detection unit detects information relating to the collision of objects in a scene, and the haptics propagation characteristic setting unit sets the haptics propagation characteristics based on the collision state indicated by the detected value. (10) The decoding device according to any one of (7) to (9), wherein the detection unit detects information relating to the contact state between a haptics device and a user in real space, and the haptics propagation characteristic setting unit sets the haptics propagation characteristics based on the contact state indicated by the detected value. (11) The decoding device according to any one of (1) to (10), further comprising a detection unit that detects predetermined information relating to the external environment, wherein the receptor haptics data generation unit generates the receptor haptics data based on the detected value. (12) The decoding device according to (11), wherein the detection unit detects information regarding the contact state between an object and a user in a scene, and the receptor haptics data generation unit generates the receptor haptics data based on the contact state indicated by the detected value. (13) The decoding device according to (11) or (12), wherein the detection unit detects information regarding the collision of objects in a scene, and the receptor haptics data generation unit generates the receptor haptics data based on the collision state indicated by the detected value. (14) The decoding device according to (11), wherein the detection unit detects information regarding the contact state between a haptics device and a user in real space, and the receptor haptics data generation unit generates the receptor haptics data based on the contact state indicated by the detected value. (15) The decoding device according to any one of (1) to (14), wherein the receptor haptics data generation unit generates the receptor haptics data using the haptics propagation characteristics that indicate the propagation characteristics of haptics data by avatar.(16) The decoding device according to any one of (1) to (15), wherein the receptor haptics data generation unit generates a plurality of receptor haptics data propagated along different haptics propagation paths, and mixes the generated plurality of receptor haptics data to generate mixed receptor haptics data. (17) The decoding device according to any one of (1) to (16), further comprising a haptics propagation characteristic convolution unit that derives a convolutional haptics propagation characteristic by convolving a plurality of haptics propagation characteristics, wherein the receptor haptics data generation unit generates the receptor haptics data using the derived convolutional haptics propagation characteristic. (18) The decoding device according to (17), wherein the haptics propagation characteristic convolution unit holds the derived convolutional haptics propagation characteristic, and the receptor haptics data generation unit generates the receptor haptics data using the held convolutional haptics propagation characteristic. (19) The decoding device according to (17) or (18), wherein the haptics propagation characteristic convolution unit holds the plurality of haptics propagation characteristics, derives the convolutional haptics propagation characteristics by convolving the plurality of held haptics propagation characteristics according to the propagation path, and the receptor haptics data generation unit generates the receptor haptics data using the derived convolutional haptics propagation characteristics. (20) The decoding device according to any one of (1) to (19), further comprising a scene description acquisition unit that acquires a scene description including haptics-related information relating to haptics, wherein the receptor haptics data generation unit generates the receptor haptics data based on the haptics-related information included in the acquired scene description. (21) The decoding device according to (20), wherein the haptics-related information includes information relating to the haptics propagation characteristics. (22) The decoding device according to (20) or (21), wherein the haptics-related information includes information relating to the source. (23) The decoding device according to any one of (20) to (22), wherein the haptics-related information includes information relating to the receptor.(24) The decoding device according to any one of (1) to (23), further comprising a haptics propagation path length derivation unit for deriving the haptics propagation path length, wherein the receptor haptics data generation unit generates the receptor haptics data using the derived haptics propagation path length. (25) The decoding device according to any one of (1) to (24), further comprising a receptor haptics data output unit for outputting the generated receptor haptics data. (26) A decoding method comprising decoding a bitstream to generate source haptics data which is haptics data at a source, and generating receptor haptics data which is haptics data sensed by a receptor using the source haptics data, the haptics propagation path length which is the path length of the propagation path of the source haptics data from the source to the receptor, and haptics propagation characteristics which indicate the propagation characteristics of the haptics data by the haptics propagation medium constituting the propagation path. (27) A program for causing a computer to perform a process that includes decoding a bitstream to generate source haptic data, which is haptic data at the source; and generating receptor haptic data, which is haptic data sensed by the receptor, using the source haptic data, the haptic propagation path length, which is the path length of the propagation path of the source haptic data from the source to the receptor, and the haptic propagation characteristics, which indicate the propagation characteristics of the haptic data by the haptic propagation medium constituting the propagation path.

[0373] (31) An encoding device comprising: a scene description generation unit that generates a scene description and stores haptics-related information relating to haptics in the generated scene description; and a haptics encoding unit that encodes source haptics data which is haptics data at the source. (32) The encoding device according to (31), wherein the haptics-related information includes information defining objects relating to haptics in the scene. (33) The encoding device according to (31) or (32), wherein the haptics-related information includes information relating to haptics propagation characteristics that indicate the propagation characteristics of haptics data by a haptics propagation medium that constitutes the propagation path of source haptics data from the source to the receptor. (34) The encoding device according to (33), wherein the information relating to haptics propagation characteristics includes identification information of the haptics propagation characteristics. (35) The encoding device according to (33) or (34), wherein the information relating to haptics propagation characteristics includes information relating to operation settings when the renderer is not supported. (36) The encoding device according to any one of (33) to (35), wherein the information relating to the haptic propagation characteristics includes information relating to the type of the haptic propagation characteristics. (37) The encoding device according to any one of (33) to (36), wherein the information relating to the haptic propagation characteristics includes information relating to the property definition of the haptic propagation characteristics. (38) The encoding device according to any one of (33) to (37), wherein the information relating to the haptic propagation characteristics includes information relating to the propagation delay of the haptic data. (39) The encoding device according to any one of (31) to (38), wherein the haptic-related information includes information relating to the source. (40) The encoding device according to (39), wherein the information relating to the source includes identification information of the source. (41) The encoding device according to (39) or (40), wherein the information relating to the source includes information relating to the type of the source. (42) The encoding device according to (39), wherein the information relating to the source includes information relating to the pre-gain of the source.(43) The encoding apparatus according to any one of (39) to (42), wherein the information relating to the source includes information relating to the response during audio playback. (44) The encoding apparatus according to any one of (39) to (43), wherein the information relating to the source includes information relating to the specification of the attenuation function. (45) The encoding apparatus according to any one of (39) to (44), wherein the information relating to the source includes information relating to parameters applied to the attenuation function. (46) The encoding apparatus according to any one of (39) to (45), wherein the information relating to the source includes information relating to the distance at which the gain becomes 1. (47) The encoding apparatus according to any one of (39) to (46), wherein the information relating to the source includes information relating to an accessor. (48) The encoding apparatus according to any one of (39) to (47), wherein the information relating to the source includes information relating to the presence or absence of air propagation. (49) The encoding apparatus according to any one of (31) to (48), wherein the haptics-related information includes information relating to a receptor. (50) The encoding device according to (49), wherein the information relating to the receptor includes identification information of the receptor. (51) The encoding device according to any one of (31) to (50), further comprising a scene description supply unit that supplies the generated scene description. (52) An encoding method comprising generating a scene description, storing haptics-related information relating to haptics in the generated scene description, and encoding source haptics data which is haptics data at a source. (53) A program for causing a computer to perform a process comprising generating a scene description, storing haptics-related information relating to haptics in the generated scene description, and encoding source haptics data which is haptics data at a source.

[0374] (61) The decoding device according to (1), wherein the receptor haptics data generation unit further generates the receptor haptics data using propagation path configuration information that shows the configuration of the propagation path including branching. (62) The receptor haptics data generation unit generates first haptics data for the first propagation path and the second propagation path using the source haptics data, the first propagation path length which is the path length of the first haptics propagation medium, and the first haptics propagation characteristics of the first haptics propagation medium, based on the propagation path configuration information which includes a first propagation path from the source to a first receptor via a first haptics propagation medium and a second haptics propagation medium, and for the first propagation path and the second propagation path, the decoding device according to (1), wherein the receptor haptics data generation unit further generates the receptor haptics data using propagation path configuration information that shows the configuration of the propagation path including branching. The decoding device according to (61), wherein, for the first propagation path, first receptor haptic data sensed by the first receptor is generated using the first haptic data, a second propagation path length which is the path length of the second haptic propagation medium, and a second haptic propagation characteristic of the second haptic propagation medium; and for the second propagation path, second receptor haptic data sensed by the second receptor is generated using the first haptic data, a third propagation path length which is the path length of the third haptic propagation medium, and a third haptic propagation characteristic of the third haptic propagation medium.

[0375] (71) The decoding device according to (1), wherein the receptor haptics data generation unit further generates the receptor haptics data using propagation path control information that controls the use of the haptics propagation medium. (72) The decoding device according to (71), wherein the propagation path control information indicates whether or not to use the haptics propagation medium, and the receptor haptics data generation unit generates the receptor haptics data for the propagation path composed of the haptics propagation medium that is indicated to be used by the propagation path control information. (73) The decoding device according to (71) or (71), wherein the propagation path control information indicates the priority of use of the haptics propagation medium, and the receptor haptics data generation unit generates the receptor haptics data for the propagation path composed of the haptics propagation medium that has a higher priority set in the propagation path control information. (74) The propagation path control information further indicates a threshold for the difference in levels of the receptor haptics data, and the receptor haptics data generation unit blocks the propagation path by not using the haptics propagation medium that constitutes the propagation path of the second receptor haptics data, and erases the second receptor haptics data, according to any one of (71) to (73). (75) The propagation path control information further controls the use of the source, and the receptor haptics data generation unit generates the receptor haptics data corresponding to the source based on the propagation path control information, according to any one of (71) to (74).

[0376] (81) The decoding device according to (1), wherein the receptor haptics data generation unit further generates the receptor haptics data sensed by the receptor using receptor control information that controls the use of the receptor. (82) The decoding device according to (81), wherein the receptor control information indicates whether or not to use the receptor, and the receptor haptics data generation unit generates the receptor haptics data sensed by the receptor that is indicated to be used by the receptor control information. (83) The decoding device according to (81) or (82), wherein the receptor control information indicates the sensitivity of the receptor, and the receptor haptics data generation unit generates the receptor haptics data sensed by the receptor using the sensitivity indicated by the receptor control information.

[0377] (91) The decoding device according to (1), further comprising a haptics display unit that outputs physical motion corresponding to the receptor haptics data sensed by the receptor associated with itself. (92) The decoding device according to (91), wherein the haptics display unit outputs physical motion corresponding to synthesized receptor haptics data obtained by synthesizing the receptor haptics data sensed by each of the plurality of receptors associated with itself.

[0378] (101) The decoding device according to (1), wherein the haptic propagation characteristics indicate the frequency-specific propagation characteristics of the haptic data by the haptic propagation medium constituting the propagation path.

[0379] 300 Playback device, 311 Haptics decoding unit, 312 Haptics processing unit, 313 Haptics presentation unit, 314 Input unit, 315 Sensor unit, 321 Haptics propagation path length derivation unit, 322 Receptor haptics data generation unit, 323 Haptics propagation characteristic setting unit, 500 File generation device, 511 SD generation unit, 512 SD encoding unit, 513 Haptics encoding unit, 514 Storage unit, 515 Supply unit, 600 Playback device, 611 MAF, 612 Buffer, 613 PE, 614 Sensor unit, 615 Input unit, 616 Haptics presentation unit, 621 Content file acquisition unit, 622 Haptics decoding unit, 631 SD acquisition unit, 632 SD analysis unit, 633 Haptics processing unit, 1900 Computer

Claims

1. A decoding device comprising: a decoding unit that decodes a bitstream to generate source haptic data, which is haptic data at the source; and a receptor haptic data generation unit that generates receptor haptic data, which is haptic data sensed by the receptor, using the source haptic data, the haptic propagation path length, which is the path length of the propagation path of the source haptic data from the source to the receptor, and haptic propagation characteristics, which indicate the propagation characteristics of the haptic data by the haptic propagation medium constituting the propagation path.

2. The decoding device according to claim 1, wherein the haptic propagation medium is space, and the receptor haptic data generation unit generates the receptor haptic data using the source haptic data, an arbitrary haptic propagation path length in the space, and the haptic propagation characteristics based on the components or state of the space.

3. The decoding device according to claim 1, wherein the haptic propagation medium is an object, and the receptor haptic data generation unit generates the receptor haptic data using the source haptic data, the haptic propagation path length according to the shape of the object, and the haptic propagation characteristics according to the material, shape, or state of the object.

4. The decoding device according to claim 1, further comprising a haptic propagation characteristic setting unit for setting the haptic propagation characteristics, wherein the receptor haptic data generation unit generates the receptor haptic data using the set haptic propagation characteristics.

5. The decoding device according to claim 4, wherein the haptic propagation characteristic setting unit selects a function to be used as the haptic propagation characteristic from among a plurality of candidates.

6. The decoding device according to claim 4, wherein the haptic propagation characteristic setting unit sets the coefficients of a predetermined model of the haptic propagation characteristics.

7. The decoding device according to claim 1, further comprising: a detection unit for detecting predetermined information relating to the external environment; and a haptic propagation characteristic setting unit for setting the haptic propagation characteristics based on the detected values.

8. The decoding device according to claim 7, wherein the detection unit detects information regarding the contact state between an object and a user in a scene, and the haptics propagation characteristic setting unit sets the haptics propagation characteristics based on the contact state indicated by the detected value.

9. The decoding device according to claim 7, wherein the detection unit detects information regarding the collision of objects in a scene, and the haptic propagation characteristic setting unit sets the haptic propagation characteristics based on the collision indicated by the detected value.

10. The decoding device according to claim 7, wherein the detection unit detects information regarding the contact state between the haptic device and the user in real space, and the haptic propagation characteristic setting unit sets the haptic propagation characteristics based on the contact state indicated by the detected value.

11. The decoding device according to claim 1, further comprising a detection unit for detecting predetermined information relating to the external environment, wherein the receptor haptics data generation unit generates the receptor haptics data based on the detected values.

12. The decoding device according to claim 1, wherein the receptor haptics data generation unit generates the receptor haptics data using the haptics propagation characteristics that show the propagation characteristics of haptics data by avatars.

13. The decoding device according to claim 1, wherein the receptor haptics data generation unit generates a plurality of receptor haptics data propagated along different haptics propagation paths, and mixes the generated plurality of receptor haptics data to generate mixed receptor haptics data.

14. The decoding device according to claim 1, further comprising a haptic propagation characteristic convolution unit that derives a convolutional haptic propagation characteristic by convolving a plurality of haptic propagation characteristics, wherein the receptor haptic data generation unit generates the receptor haptic data using the derived convolutional haptic propagation characteristic.

15. The decoding device according to claim 14, wherein the haptic propagation characteristic convolution unit holds the derived convolutional haptic propagation characteristics, and the receptor haptic data generation unit generates the receptor haptic data using the held convolutional haptic propagation characteristics.

16. The decoding device according to claim 14, wherein the haptic propagation characteristic convolution unit holds the plurality of haptic propagation characteristics, derives the convolutional haptic propagation characteristics by convolving the plurality of held haptic propagation characteristics according to the propagation path, and the receptor haptic data generation unit generates the receptor haptic data using the derived convolutional haptic propagation characteristics.

17. The decoding device according to claim 1, further comprising a scene description acquisition unit that acquires a scene description including haptics-related information relating to haptics, wherein the receptor haptics data generation unit generates the receptor haptics data based on the haptics-related information included in the acquired scene description.

18. The decoding device according to claim 17, wherein the haptics-related information includes information relating to the haptics propagation characteristics.

19. The decoding device according to claim 1, wherein the receptor haptics data generation unit further generates the receptor haptics data using propagation path configuration information that shows the configuration of the propagation path including branching.

20. The receptor haptics data generation unit generates first haptics data for the first propagation path and the second propagation path using the source haptics data, the first propagation path length which is the path length of the first haptics propagation medium, and the first haptics propagation characteristics of the first haptics propagation medium, based on the propagation path configuration information which includes a first propagation path from the source to the first receptor via a first haptics propagation medium and a second haptics propagation medium, and a second propagation path from the source to the second receptor via the first haptics propagation medium. The decoding device according to claim 19, wherein, with respect to the first propagation path, first receptor haptic data sensed by the first receptor is generated using the first haptic data, a second propagation path length which is the path length of the second haptic propagation medium, and a second haptic propagation characteristic of the second haptic propagation medium; and with respect to the second propagation path, second receptor haptic data sensed by the second receptor is generated using the first haptic data, a third propagation path length which is the path length of the third haptic propagation medium, and a third haptic propagation characteristic of the third haptic propagation medium.

21. The decoding device according to claim 1, wherein the receptor haptics data generation unit further generates the receptor haptics data using propagation path control information that controls the use of the haptics propagation medium.

22. The decoding device according to claim 21, wherein the propagation path control information indicates whether or not to use the haptic propagation medium, and the receptor haptic data generation unit generates the receptor haptic data for the propagation path which is composed of the haptic propagation medium that is indicated to be used by the propagation path control information.

23. The decoding device according to claim 21, wherein the propagation path control information indicates the priority of using the haptic propagation medium, and the receptor haptic data generation unit generates the receptor haptic data for the propagation path which is composed of the haptic propagation medium for which a higher priority has been set in the propagation path control information.

24. The decoding device according to claim 21, wherein the propagation path control information further indicates a threshold for the difference in levels of the receptor haptics data, and the receptor haptics data generation unit, when the level of the first receptor haptics data is greater than the threshold indicated by the propagation path control information compared to the level of the second receptor haptics data, blocks the propagation path by not using the haptics propagation medium that constitutes the propagation path of the second receptor haptics data, and erases the second receptor haptics data.

25. The decoding device according to claim 21, wherein the propagation path control information further controls the use of the source, and the receptor haptics data generation unit generates the receptor haptics data corresponding to the source based on the propagation path control information.

26. The decoding device according to claim 1, wherein the receptor haptics data generation unit further generates the receptor haptics data sensed by the receptor using receptor control information that controls the use of the receptor.

27. The decoding device according to claim 26, wherein the receptor control information indicates whether or not to use the receptor, and the receptor haptics data generation unit generates the receptor haptics data sensed by the receptor that is indicated to be used by the receptor control information.

28. The decoding device according to claim 26, wherein the receptor control information indicates the sensitivity of the receptor, and the receptor haptics data generation unit generates the receptor haptics data sensed by the receptor using the sensitivity indicated by the receptor control information.

29. The decoding device according to claim 1, further comprising a haptics presentation unit that outputs physical motion corresponding to the receptor haptics data sensed by the receptor associated with itself.

30. The decoding device according to claim 29, wherein the haptic display unit outputs physical motion corresponding to synthesized receptor haptic data obtained by synthesizing the receptor haptic data sensed by each of the plurality of receptors associated with itself.

31. The decoding device according to claim 1, wherein the haptic propagation characteristics indicate the frequency-specific propagation characteristics of the haptic data by the haptic propagation medium constituting the propagation path.

32. A decoding method comprising: decoding a bitstream to generate source haptic data, which is haptic data at the source; and generating receptor haptic data, which is haptic data sensed by the receptor, using the source haptic data, the haptic propagation path length, which is the path length of the propagation path of the source haptic data from the source to the receptor, and haptic propagation characteristics, which indicate the propagation characteristics of the haptic data by the haptic propagation medium constituting the propagation path.

33. An encoding method comprising generating a scene description, storing haptics-related information concerning haptics in the generated scene description, and encoding source haptics data which is haptics data at the source.

Images