Method and apparatus for modeling objects in a virtual environment and providing haptic feedback responsive to interaction

Abstract

A method, apparatus, and computer program product are provided for modeling object behaviors and providing haptic signal mediation through energy propagation in virtual environments. Methods for rendering haptic feedback in a virtual environment can include encoding energy interactions with one or more virtual objects within a virtual scene based on object properties of the one or more virtual objects, interactions with the one or more virtual objects, and energy transformations associated with the one or more virtual objects; propagating energy across the one or more virtual objects; updating energy states of the one or more virtual objects; and mapping the energy interactions of the one or more virtual objects to energy outputs for one or more haptic devices based at least in part on the energy states of the one or more virtual objects.

Description

METHOD AND APPARATUS FOR MODELING OBJECTS IN A VIRTUAL ENVIRONMENT AND PROVIDING HAPTIC FEEDBACK RESPONSIVE TO INTERACTIONTECHNOLOGICAL FIELD

[0001] Certain example embodiments of the present disclosure relates to modeling objects in a virtual environment, and more particularly, to modeling object behaviors and providing haptic signal mediation through energy propagation in virtual environments.BACKGROUND

[0002] Virtual Reality (VR) systems and wearable devices configured to provide feedback to a user relating to the virtual environment have varying degrees of capability. The information can be rendered in ways that can be perceived by any of the human senses to provide realistic and immersive experiences. While some devices are capable of providing tactile feedback through vibrational energy, kinetic energy, thermal energy, etc., other devices can be limited in their capabilities.BRIEF SUMMARY

[0003] A method, apparatus, and computer program product are therefore provided for modeling objects in a virtual environment, and more particularly, to modeling object behaviors and providing haptic signal mediation through energy propagation in virtual environments. Certain embodiments provided herein include an apparatus including at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least: identify one or more virtual objects in a virtual scene; determine object properties for the one or more virtual objects; calculate, for the one or more virtual objects, energy transformations; encode energy interactions for the one or more virtual objects based on the object properties and the energy transformations; and map the energy interactions for the one or more virtual objects to energy output for one or more haptic devices.

[0004] According to some embodiments, causing the apparatus to determine object properties for the one or more virtual objects includes causing the apparatus to: assign an energy signature to the one or more virtual objects, where the energy signature is associated with a position of the one or more virtual objects within the virtual scene, a proximity of the one or more virtual objects to one or more other virtual objects within the virtual scene, and an interaction between the one or more virtual objects and the one or more other virtual objects within the scene. According to some embodiments the energy interactions include one or more of kinetic energy, thermal energy, vibrational energy, potential energy, chemical energy or electrical energy.

[0005] Causing the apparatus of an example embodiment to map the energy interactions for the one or more virtual objects to energy output for one or more haptic devices includes causing the apparatus to: determine supported functions of the one or more haptic devices; and map the energy interactions for the one or more virtual objects to the supported functions. According to some embodiments, causing the apparatus to map the energy interactions for the one or more virtual objects to energy output for one or more haptic devices includes causing the apparatus to: convert the energy interactions from at least one of kinetic energy, thermal energy, vibrational energy, potential energy, chemical energy, or electrical energy to at least one other of kinetic energy, thermal energy, vibrational energy, potential energy, chemical energy or electrical energy based on the supported functions.

[0006] Causing the apparatus of an example embodiment to calculate, for the one or more virtual objects, energy transformations includes causing the apparatus to: calculate energy transformations for the one or more virtual objects based, at least in part, on object properties of the one or more virtual objects, wherein the object properties comprise one or more of density, thermal conductivity, coefficient of restitution, or a spring constant. The apparatus of some embodiments is further caused to: calibrate the energy output to an energy output range for the one or more haptic devices based on identified capabilities of the one or more haptic devices.

[0007] According to some embodiments, causing the apparatus to calibrate the energy output to an energy output range for the one or more haptic devices based on identified capabilities of the one or more haptic devices includes causing the apparatus to: identify the identified capability of the one or more haptic devices based on an initialization of at least one of the virtual scene or the one or more haptic devices. Causing the apparatus of some embodiments todetermine the object properties for the one or more virtual objects includes causing the apparatus to: determine time-dependent object properties; and attenuate the time-dependent object properties over time.

[0008] Causing the apparatus of some embodiments to attenuate the time-dependent object properties over time includes causing the apparatus to: determine that a user is not interacting with the one or more virtual objects; and reduce a frequency at which the time-dependent object properties are attenuated over time in response to determining that the user is not interacting with the one or more virtual objects. According to some embodiments, causing the apparatus to attenuate the time-dependent object properties over time includes causing the apparatus to: determine that a user is interacting with the one or more virtual objects; and increase a frequency at which the time-dependent object properties are attenuated over time in response to determining that the user is interacting with the one or more virtual objects. The apparatus of some embodiments is further caused to: synchronize the virtual scene based on a rendering frequency of the one or more haptic devices.

[0009] Certain embodiments provided herein include a method including: identifying one or more virtual objects in a virtual scene; determining object properties for the one or more virtual objects; calculating, for the one or more virtual objects, energy transformations; encoding energy interactions for the one or more virtual objects based on the object properties and the energy transformations; and mapping the energy interactions for the one or more virtual objects to energy output for one or more haptic devices. According to some embodiments, determining object properties for the one or more virtual objects includes: assigning an energy signature to the one or more virtual objects, wherein the energy signature is associated with a position of the one or more virtual objects within the virtual scene, a proximity of the one or more virtual objects to one or more other virtual objects within the virtual scene, and an interaction between the one or more virtual objects and the one or more other virtual objects within the scene.

[0010] According to certain embodiments the energy interactions include one or more of kinetic energy, thermal energy, vibrational energy, potential energy, chemical energy or electrical energy. According to some embodiments mapping the energy interactions for the one or more virtual objects to energy output for one or more haptic devices includes: determining supported functions of the one or more haptic devices; and mapping the energy interactions for the one or more virtual objects to the supported functions. Mapping the energy interactions forthe one or more virtual objects to energy output for one or more haptic devices includes in some embodiments: converting the energy interactions from at least one of kinetic energy, thermal energy, vibrational energy, potential energy, chemical energy, or electrical energy to at least one other of kinetic energy, thermal energy, vibrational energy, potential energy, chemical energy or electrical energy based on the supported functions.

[0011] According to some embodiments calculating, for the one or more virtual objects, energy transformations includes: calculating energy transformations for the one or more virtual objects based, at least in part, on object properties of the one or more virtual objects, wherein the object properties comprise one or more of density, thermal conductivity, coefficient of restitution, or a spring constant. The method of some embodiments further includes calibrating the energy output to an energy output range for the one or more haptic devices based on identified capabilities of the one or more haptic devices. According to some embodiments calibrating the energy output to an energy output range for the one or more haptic devices based on identified capabilities of the one or more haptic devices includes: identifying the identified capability of the one or more haptic devices based on an initialization of at least one of the virtual scene or the one or more haptic devices.

[0012] Determining the object properties for the one or more virtual objects includes, in some embodiments: determining time-dependent object properties; and attenuating the timedependent object properties over time. According to certain embodiments attenuating the timedependent object properties over time includes: determining that a user is not interacting with the one or more virtual objects; and reducing a frequency at which the time-dependent object properties are attenuated over time in response to determining that the user is not interacting with the one or more virtual objects.

[0013] According to certain embodiments, attenuating the time-dependent object properties over time includes: determining that a user is interacting with the one or more virtual objects; and increasing a frequency at which the time-dependent object properties are attenuated over time in response to determining that the user is interacting with the one or more virtual objects. The method of some embodiments further includes synchronizing the virtual scene based on a rendering frequency of the one or more haptic devices.

[0014] According to some embodiments encoding energy interactions for the one or more virtual objects based on the object properties and the energy transformations includes: encodingenergy interactions for a first of the one or more virtual objects interacting with a second of the one or more virtual objects, wherein an energy transformation occurs with energy from the first of the one or more virtual objects to energy of the second of the one or more virtual objects. According to certain embodiments the energy transformation with energy from the first of the one or more virtual objects to energy of the second of the one or more virtual objects is propagated based on properties of a medium of the first of the one or more virtual objects and properties of a medium of the second of the one or more virtual objects. According to certain embodiments encoding energy interactions for the one or more virtual objects based on the object properties and the energy transformations includes: encoding energy interactions for a user interacting with a first of the one or more virtual objects, wherein an energy transformation occurs with energy from the user to energy of the one or more virtual objects.

[0015] Certain embodiments provided herein include a method for rendering haptic feedback in a virtual environment including: encoding energy interactions with one or more virtual objects within a virtual scene based on object properties of the one or more virtual objects, interactions with the one or more virtual objects, and energy transformations associated with the one or more virtual objects; propagating energy across the one or more virtual objects; updating energy states of the one or more virtual objects; and mapping the energy interactions of the one or more virtual objects to energy outputs for one or more haptic devices based at least in part on the energy states of the one or more virtual objects.

[0016] According to some embodiments, propagating energy across the one or more virtual objects includes: propagating energy from a first object of the one or more virtual objects to a second object of the one or more virtual objects, wherein the energy from the first object to the second object is propagated at least one of directly between the first object and the second object or indirectly through an element of the virtual scene. According to certain embodiments updating the energy states of the one or more virtual objects is performed at a frequency proportional to user interaction with the one or more virtual objects.

[0017] Updating the energy states of the one or more virtual objects in an example embodiment is performed at a frequency based on an output frequency of the one or more haptic devices. Mapping the energy interactions of the one or more virtual objects to energy outputs for the one or more haptic devices is performed, in some embodiments, based on supported energy outputs for the one or more haptic devices. According to some embodiments the energyinteractions include one or more of kinetic energy, thermal energy, vibrational energy, potential energy, or electrical energy.

[0018] Mapping the energy interactions of the one or more virtual objects to the energy outputs for one or more haptic devices includes, in some embodiments: converting the energy interactions from at least one of kinetic energy, thermal energy, vibrational energy, potential energy, or electrical energy to at least one other of kinetic energy, thermal energy, vibrational energy, potential energy, chemical energy, or electrical energy based on supported functions of the one or more haptic devices. The method of some embodiments further includes: calibrating the energy outputs to energy output ranges for the one or more haptic devices based on identified capabilities of the one or more haptic devices.

[0019] The method of some embodiments further includes: determining time-dependent object properties; and attenuating the time-dependent object properties over time. Attenuating the time-dependent object properties over time includes, in some embodiments: determining that a user is not interacting with the one or more virtual objects; and reducing a frequency at which the time-dependent object properties are attenuated over time in response to determining that the user is not interacting with the one or more virtual objects. According to some embodiments attenuating the time-dependent object properties over time includes: determining that a user is interacting with the one or more virtual objects; and increasing a frequency at which the timedependent object properties are attenuated over time in response to determining that the user is interacting with the one or more virtual objects. The method of some embodiments further includes synchronizing the virtual scene based on a rendering frequency of the one or more haptic devices.

[0020] Certain embodiments provided herein include an apparatus including at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least: encode energy interactions with one or more virtual objects within a virtual scene based on object properties of the one or more virtual objects, interactions with the one or more virtual objects, and energy transformations associated with the one or more virtual objects; propagate energy across the one or more virtual objects; update energy states of the one or more virtual objects; and map the energy interactions of the one or more virtual objects to energyoutputs for one or more haptic devices based at least in part on the energy states of the one or more virtual objects.

[0021] Causing the apparatus of some embodiments to propagate energy across the one or more virtual objects includes causing the apparatus to: propagate energy from a first object of the one or more virtual objects to a second object of the one or more virtual objects, where the energy from the first object to the second object is propagated at least one of directly between the first object and the second object or indirectly through an element of the virtual scene. According to some embodiments updating the energy states of the one or more virtual objects is performed at a frequency proportional to user interaction with the one or more virtual objects. According to certain embodiments updating the energy states of the one or more virtual objects is performed at a frequency based on an output frequency of the one or more haptic devices.

[0022] Mapping the energy interactions of the one or more virtual objects to energy outputs for the one or more haptic devices is performed, in some embodiments, based on supported energy outputs for the one or more haptic devices. The energy interactions of some embodiments include one or more of kinetic energy, thermal energy, vibrational energy, potential energy, or electrical energy. Causing the apparatus of some embodiments to map the energy interactions of the one or more virtual objects to the energy outputs for one or more haptic devices includes causing the apparatus to: convert the energy interactions from at least one of kinetic energy, thermal energy, vibrational energy, potential energy, or electrical energy to at least one other of kinetic energy, thermal energy, vibrational energy, potential energy, chemical energy, or electrical energy based on supported functions of the one or more haptic devices.

[0023] According to some embodiments the apparatus is further caused to: calibrate the energy outputs to energy output ranges for the one or more haptic devices based on identified capabilities of the one or more haptic devices. According to certain embodiments the apparatus is further caused to: determine time-dependent object properties; and attenuate the time-dependent object properties over time.

[0024] Causing the apparatus of some embodiments to attenuate the time-dependent object properties over time includes causing the apparatus to: determine that a user is not interacting with the one or more virtual objects; and reduce a frequency at which the time-dependent object properties are attenuated over time in response to determining that the user is not interacting with the one or more virtual objects. According to some embodiments causing the apparatus toattenuate the time-dependent object properties over time includes causing the apparatus to: determine that a user is interacting with the one or more virtual objects; and increase a frequency at which the time-dependent object properties are attenuated over time in response to determining that the user is interacting with the one or more virtual objects. The apparatus of some embodiments is further configured to: synchronize the virtual scene based on a rendering frequency of the one or more haptic devices.

[0025] Certain embodiments provided herein include an apparatus including: means for identifying one or more virtual objects in a virtual scene; means for determining object properties for the one or more virtual objects; means for calculating, for the one or more virtual objects, energy transformations; means for encoding energy interactions for the one or more virtual objects based on the object properties and the energy transformations; and means for mapping the energy interactions for the one or more virtual objects to energy output for one or more haptic devices. According to some embodiments, the means for determining object properties for the one or more virtual objects includes: means for assigning an energy signature to the one or more virtual objects, where the energy signature is associated with a position of the one or more virtual objects within the virtual scene, a proximity of the one or more virtual objects to one or more other virtual objects within the virtual scene, and an interaction between the one or more virtual objects and the one or more other virtual objects within the scene.

[0026] According to another aspect of the disclosure, there is provided a non-transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform a method in accordance with one or more aspects described herein.

[0027] According to another aspect of the disclosure, there is provided an apparatus comprising means for performing, or causing the apparatus to perform, a method in accordance with one or more aspects described herein.

[0028] The above summary is provided merely for purposes of summarizing certain example embodiments of the disclosure so as to provide a basic understanding of some aspects of the disclosure. Accordingly, it will be appreciated that the above described example embodiments are merely examples and should not be construed to narrow the scope or spirit of the disclosure in any way. It will be appreciated that the scope of the disclosure encompasses many potentialembodiments, some of which will be further described below, in addition to those here summarized.BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Having thus described certain example embodiments of the present disclosure in general terms, reference will hereinafter be made to the accompanying drawings which are not necessarily drawn to scale, and wherein:

[0030] Figure 1 is an overview of a system that can be used for providing haptic feedback based on modeling objects in a virtual environment according to an example embodiment of the present disclosure;

[0031] Figure 2 is an exemplary schematic diagram of an apparatus for modeling objects in a virtual environment according to an example embodiment of the present disclosure;

[0032] Figure 3 is a flow diagram of a process for modeling object behaviors and providing haptic signal mediation through energy propagation in virtual environments according to an example embodiment of the present disclosure;

[0033] Figure 4 is a flowchart of operations that may be performed for modeling objects in a virtual environment according to an example embodiment of the present disclosure; and

[0034] Figure 5 is another flowchart of operations that may be performed for modeling objects in a virtual environment according to an example embodiment of the present disclosure.DETAILED DESCRIPTION

[0035] Some embodiments of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all, embodiments of the disclosure are shown. Indeed, various embodiments of the disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like reference numerals refer to like elements throughout. As used herein, the terms “data,” “content,” “information,” and similar terms may be used interchangeably torefer to data capable of being transmitted, received and / or stored in accordance with the present disclosure. Thus, use of any such terms should not be taken to limit the spirit and scope of the present disclosure.

[0036] Additionally, as used herein, the term ‘circuitry’ refers to (a) hardware-only circuit implementations (e.g., implementations in analog circuitry and / or digital circuitry); (b) combinations of circuits and computer program product(s) comprising software and / or firmware instructions stored on one or more computer readable memories that work together to cause an apparatus to perform one or more functions described herein; and (c) circuits, such as, for example, a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation even if the software or firmware is not physically present. This definition of ‘circuitry’ applies to all uses of this term herein, including in any claims. As a further example, as used herein, the term ‘circuitry’ also includes an implementation comprising one or more processors and / or portion(s) thereof and accompanying software and / or firmware. As another example, the term ‘circuitry’ as used herein also includes, for example, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, other network device, field programmable gate array, and / or other computing device.

[0037] As defined herein, a “computer-readable storage medium,” which refers to a physical storage medium (e.g., volatile or non-volatile memory device), may be differentiated from a “computer-readable transmission medium,” which refers to an electromagnetic signal.

[0038] Virtual reality, as described herein, includes a fully immersive digital environment in which a user can interact with the environment and objects therein. As described herein, virtual reality can include augmented reality and mixed reality as they each include virtual elements. Augmented reality overlays digital / virtual elements onto the real world, while mixed reality blends the real and virtual worlds together allowing for interaction between virtual objects and physical objects. Certain embodiments described herein will reference “virtual reality”; however, the term is interpreted to encompass augmented reality and mixed reality as each of these “realities” includes some degree of virtual object presence and interaction. Thus, as will be appreciated by one of ordinary skill in the art, “virtual reality” as described herein encompasses augmented reality and mixed reality, along with a fully digital virtual environment of a totally virtual reality.

[0039] Certain embodiments described herein relate to modeling of objects within a virtual or semi-virtual environment. More particularly, some embodiments relate to modeling object behaviors based on their properties and providing haptic signal mediation through energy propagation in virtual environments. At least some embodiments generate and render haptic feedback based on energy calculations adapted to energy-based models.

[0040] Haptic feedback from virtual environments helps a user feel immersed in the virtual reality; however, there remains a disconnect between a virtual environment and accurate portrayal of virtual objects and interactions therewith within the virtual environment. Certain embodiments described herein propose energy-based modeling of virtual reality interactions within a closed system. Some embodiments employ a unifying holistic approach to modeling, rendering, and mediating haptic feedback irrespective of the underlying technology or output (e.g. haptic) devices by proposing a method of using energy conservation within a virtual environment.

[0041] Certain embodiments of the present disclosure model a dynamic virtual environment and objects within he virtual environment, with or without user interaction, using energy conservation and translation to mathematically model object movements and resulting haptic outputs. Moreover, using energy translation and mediation within the environment, haptic feedback is encoded and rendered to connected haptic output devices mapping the forces and energy translation from the virtual environment to the physical devices.

[0042] Provided herein is a unified framework for modeling energy transfers within a wide range of virtual environments by accounting for mediation, attenuation, and integration of feedback signals as energy is translated within the virtual scene, thereby providing realistic and immersive output feedback irrespective of the haptic device(s) connected to the interaction. The modeling techniques and subsequent device agnostic algorithm translate energy in a virtual scene by calculation of haptic feedback signals and designing adaptive rendering techniques that optimize haptic feedback based on device-specific capabilities, thereby optimizing interaction irrespective of device limitations.

[0043] Figure 1 is an overview of a system that can be used to practice certain embodiments described herein and should not be considered limiting. As illustrated in Figure 1, certain example embodiments may be implemented as or employed in a distributed system. The various depicted components may be configured to communicate over a network, such as the Internet,for example, or any other communication interface as described in further detail hereinafter. In general, a haptics controller device 32 may be configured to communicate with a server 40 and communicate with haptic feedback devices 30, configured to render a virtual environment and provide simulated virtual effects to a user.

[0044] According to certain embodiments, haptic feedback device 30 can include any device configured to provide haptic feedback. A haptic feedback device 30 may include a wearable device such as but not limited to a head- worn display, a watch, a glove, a vest, and / or the like. According to certain embodiments, haptic feedback device 30 may comprise or incorporate any number of vibrotactile actuators, pneumatic systems, and / or force feedback systems configured to provide haptic feedback. A haptic feedback device 30 may comprise or incorporate a thermal feedback device, such as but not limited to a Peltier device, an infrared source, a heating and / or ventilating system, and / or the like, configured to control thermal energy and enable a user to perceive thermal properties. In this regard, according to certain embodiments, a virtual reality system may not necessarily include thermal systems, but other types of haptic actuators can be configured to provided haptic feedback representative of thermal properties.

[0045] The haptics controller device 32 may include a user device, such as a smart phone, gaming console, personal computer, workstation, tablet, laptop, other computing device, or the like. According to certain embodiments, the haptics controller device 32 may be configured in proximity to the haptic feedback device 30. For example, a haptics controller device 32 implemented as a gaming console may communicate over a local area network or via a near field communication protocol with the haptic feedback device 30. According to certain embodiments, although not depicted in Figure 1 as such, the haptic feedback device 30 and haptics controller device 32 can be implemented within the same device.

[0046] According to certain example embodiments, the haptics controller device 32 receives data from server 40, decodes and / or processes the data, and communicates data to the haptic feedback device 30, enabling the haptic feedback device to render haptic feedback.

[0047] Haptic feedback device 30 and / or haptics controller device 32 may further include any number of sensors, facilitating motion tracking, and / or detection of user interactions with a virtual environment. In this regard, the haptics controller device 32 can be configured to communicate user input data to the server 40.

[0048] According to certain example embodiments, any number of haptic controller devices 32 may be configured to communicate with server 40. Server 40 can be configured in proximity to the one or more devices 30, and / or remotely from the one or more devices 30. Server 40 may include any number of computing devices and may be configured to update a scene description representative of a virtual environment, and model properties of the virtual environment, including objects of the virtual environment.

[0049] The objects may be virtual objects representative of physical objects in a scene. According to certain embodiments, the server 40 is configured to communicate data to any number of haptics controller device 32 to further enable a haptics feedback device 30 to render the virtual environment. According to certain example embodiments, the server 40 and one or more devices 30 communicate in real-time or near real-time such that the server 40 updates the scene description in real-time or near real-time as user feedback and / or user input is received. Reference to near real-time is made to account for short delays in computer processing time and communication over a network, which may be nominal or not noticeable to a user.

[0050] According to certain example embodiments, the server 40 can be implemented within a same device as the haptics controller device 32. Configuration of the haptics controller device 32 and / or server 40 may be based on the use case, scalability, interoperability, performance targets, and / the like. For example, such as according to a single-user use case, the scene description can be updated at the haptics controller device 32. According to certain example embodiments, the scene description may be updated at the server 40 and may therefore facilitate multi-user use cases.

[0051] The system of Figure 1 described above is provided merely as an example implementation and it will be appreciated that certain example embodiments provided herein may be implemented as or employed by any number of system architectures. Accordingly, certain operations can be performed at the haptic feedback device 30, certain operations can be performed at the haptics controller device 32, and certain operations can be performed at the server 40.

[0052] Referring now to Figure 2, apparatus 200 is a computing device(s) configured to implement certain example embodiments disclosed herein, and to perform certain operations described herein. Apparatus 200 may at least partially or wholly embody device 30 and / or server 40.

[0053] Apparatus 200 may include or otherwise be in communication with processor 220, user interface 222, communication interface 224, and memory device 226. Each of these components may be present or omitted based on the device the apparatus embodies. For example, the user interface 222 may be considered optional in apparatus 200. For example, when apparatus 200 is embodied as server 40, user interface 222 may not be present.

[0054] According to certain embodiments, the processor 220 (and / or co-processors or any other processing circuitry assisting or otherwise associated with the processor 220) may be in communication with the memory device 226 via a bus for passing information among components of the apparatus 200. The memory device 226 may include, for example, one or more volatile and / or non-volatile memories. In other words, for example, the memory device 226 may be an electronic storage device (e.g., a computer readable storage medium) comprising gates configured to store data (e.g., bits) that may be retrievable by a machine (e.g., a computing device like the processor 220). The memory device 226 may be configured to store information, data, content, applications, instructions, or the like for enabling the apparatus to carry out various functions in accordance with certain example embodiments of the present disclosure. According to certain embodiments, memory device 226 is configured to store a model physical properties of a virtual environment including objects therein. The model can be updated as time progresses, as described according to certain example embodiments disclosed herein, and by instructions stored on memory device 226. The memory device 226 can be configured to buffer input data for processing by the processor 220. Additionally or alternatively, the memory device 226 could be configured to store instructions for execution by the processor 220.

[0055] According to certain embodiments, the apparatus 200 may be embodied as a chip or chip set. In other words, the apparatus 200 may comprise one or more physical packages (e.g., chips) including materials, components and / or wires on a structural assembly (e.g., a baseboard). The structural assembly may provide physical strength, conservation of size, and / or limitation of electrical interaction for component circuitry included thereon. The apparatus 200 may therefore, in some cases, be configured to implement an embodiment of the present disclosure on a single chip or as a single “system on a chip.” As such, in some cases, a chip or chipset may constitute means for performing one or more operations for providing the functionalities described herein.

[0056] The processor 220 may be embodied in a number of different ways. For example, the processor 220 may be embodied as one or more of various hardware processing means such as acoprocessor, a microprocessor, a controller, a digital signal processor (DSP), a processing element with or without an accompanying DSP, or various other processors including integrated circuits such as, for example, an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), a microcontroller unit (MCU), a hardware accelerator, a specialpurpose computer chip, or the like. As such, in some embodiments, the processor 220 may include one or more processing cores configured to perform independently. A multi-core processor may enable multiprocessing within a single physical package. Additionally or alternatively, the processor 220 may include one or more processors configured in tandem via the bus to enable independent execution of instructions, pipelining and / or multithreading.

[0057] According to certain example embodiments, the processor 220 may be configured to execute instructions stored in the memory device 226 or otherwise accessible to the processor 220. Alternatively or additionally, the processor 220 may be configured to execute hard coded functionality. As such, whether configured by hardware or software methods, or by a combination thereof, the processor 220 may represent an entity (e.g., physically embodied in circuitry) capable of performing operations according to an embodiment of the present disclosure while configured accordingly. Thus, for example, when the processor 220 is embodied as an ASIC, FPGA or the like, the processor 220 may be specifically configured hardware for conducting the operations described herein. Alternatively, as another example, when the processor 220 is embodied as an executor of software instructions, the instructions may specifically configure the processor 220 to perform the algorithms and / or operations described herein when the instructions are executed. For example, the processor 220 is configured to update the model stored on memory device 226 to reflect physical properties of a virtual environment including objects therein. However, in some cases, the processor 220 may be a processor of a specific device (e.g., a mobile terminal or network entity) configured to employ an embodiment of the present disclosure by further configuration of the processor 220 by instructions for performing the algorithms and / or operations described herein. The processor 220 may include, among other things, a clock, an arithmetic logic unit (ALU) and logic gates configured to support operation of the processor 220.

[0058] Meanwhile, the communication interface 224 may be any means such as a device or circuitry embodied in either hardware or a combination of hardware and software that is configured to receive and / or transmit data from / to a network and / or any other device or modulein communication with the apparatus 200. In this regard, the communication interface 224 may include, for example, an antenna (or multiple antennas) and supporting hardware and / or software for enabling communications with a wireless communication network. Additionally or alternatively, the communication interface 224 may include the circuitry for interacting with the antenna(s) to cause transmission of signals via the antenna(s) or to handle receipt of signals received via the antenna(s). In some environments, the communication interface 224 may alternatively or also support wired communication. As such, for example, the communication interface 224 may include a communication modem and / or other hardware / software for supporting communication via cable, digital subscriber line (DSL), universal serial bus (USB) or other mechanisms. Communication interface 224 may include a network, such as any wired or wireless communication network including a local area network (LAN), personal area network (PAN), wide area network (WAN), the Internet, an intranet, or the like, as well as any attendant hardware, software and / or firmware required to implement said networks (e.g. network routers and network switches). The communication interface 224 may be configured to facilitate communication between apparatus 200 and other devices. For example, communication interface 224 may enable communication of input data from device 30 to the server 40, rendering data from the server 40 to device 30, and other data communication.

[0059] According to certain embodiments, such as when apparatus 200 is embodied as device 30, a user interface 222 may be present. The user interface 222 may be configured to cause provision of any audible, visual, haptic or other output to the user. As such, the user interface 222 may include a display, such as one configured for displaying an image or video. Example displays which may be included in user interface 222 include a computer monitor, tablet screen, television monitor, head-worn display, other wearable display, a display or plurality of displays providing a virtual reality environment, and / or the like. User interface 222 may include thermal feedback devices that provided localized feedback, such as heat pads or cooling strips, which may be incorporated in one or more wearable devices, for example. User interface 222 may include room-scale temperate controllers, such as HVAC systems, and / or the like. User interface 222 may include vibrotactile actuators, pneumatic actuators, force feedback systems, and other haptic feedback mechanisms. According to certain embodiments, user interface 222 may include additional output mechanisms.

[0060] The user interface 222 may include a means for user input, such as a keyboard, buttons, mousejoystick, touch screen, touch areas, scroller ball, soft keys, a microphone, or other input mechanisms. In some examples, the user interface 222 may include or may be communicatively connected to any number of sensors, such as a motion sensor, accelerometer, and / or the like, configured to detect movement of a wearable device, handheld device, or other device, such that the user can provide inputs based on such movements. For example, a user may turn their head while wearing a head-worn device comprising a display to view other areas of displayed content and experience corresponding feedback pertaining to a particular area of a virtual environment. User interface 222 may include eye-tracking sensors to determine which portions of content are viewed by a user. The input mechanisms are provided as examples, and according to certain embodiments, user interface 222 may include additional output mechanisms.

[0061] According to certain example embodiments, the processor 220 may comprise user interface circuitry configured to control at least some functions of one or more user interface elements. The processor 220 and / or user interface circuitry comprising the processor 220 may be configured to control one or more functions of one or more user interface elements through computer program instructions (e.g., software and / or firmware) stored on a memory accessible to the processor 220 (e.g., memory device 226, and / or the like). According to certain example embodiments, user interface 222 may be embodied by a user device that is remote from or external to apparatus 200.

[0062] Certain embodiments described herein provide an intelligent haptic signal mediation framework to provide a novel method for encoding, simulating, and mediating energy interactions within a virtual scene to generate realistic haptic feedback using calibrated rendering device(s). A closed virtual environment is a complex dynamic system with multiple objects interacting with each other with or without direct user interaction overtime. Certain embodiments provide a holistic method of simulating, encoding, mediating and then delivering precise localized and spatial (interaction) information to the user as and when they interact with specific aspects and / or objects within the dynamic environment. At least some embodiments model energy transformations employing the first law of thermodynamics to model kinetic, potential, thermal, electrical, chemical, and magnetic energy. Further, modeling of transmission losses and medium-dependent attenuation within the closed system simulates a realistic and immersive dynamic environment that can mediate haptic signals from the source to the point of interactionY1with a user. Certain embodiments further detail how this feedback signal can be calibrated to generate (haptic feedback) energy envelope that should be communicated to the user through the attached haptic devices and their corresponding actuation technologies. Haptic devices described herein can employ some or all of vibrotactile feedback, kinesthetic feedback, pneumatic feedback, thermal feedback, electro-tactile feedback, electrostatic feedback, etc.

[0063] The framework of at least some embodiments of the present disclosure supports dynamic user inputs, allowing external energy to be exerted on virtual scene objects, while generating real-time haptic feedback. Integrating IHSM (Intelligent Haptic Signal Mediation) with existing MPEG-I (Moving Picture Experts Group - 1) and MPEG SD (Moving Picture Experts Group - Standard Definition) standards enables efficient encoding and transmission of haptic data. In the downstream rendering device, the energy bandwidth is mapped to the output threshold which ensures the relative propagation of feedback proportional to the output capabilities of the connected haptic device(s). For example, in a vibrotactile interaction device, the rendered output would be calibrated using minimum and maximum energy dissipation of the vibration actuator(s). Similarly, in a thermal or pneumatic output device, the rendered feedback would correlate to the resolution and bandwidth of the device yield. Multi-technology devices that deliver a combination of haptic outputs (e.g., thermal, vibrotactile, and kinesthetic) can deliver each energy signature from the scene or object interaction, thereby providing a realistic and immersive experience to the user(s).

[0064] Figure 3 illustrates an example embodiment of a method according to the present disclosure. Initialization 300 of the virtual scene begins with loading properties 301 of objects within the scene, calibrating devices 302 for output of haptic feedback, and detecting capabilities 303 of the haptic devices connected to the virtual scene. The scene is then ready at 305 to receive interactions from a user. At interaction detection 310 is performed to identify a user input. A collision is detected at 311, which can be a user-object collision or object-object collision within the scene. User input is identified at 312, and transformations of energy within the scene are triggered at 313. The user input that is detected at 315 is used for energy propagation 320 for the objects within the virtual scene. Attenuation is computed at 321, an adjustment is made based on the medium at 322, and the energy transfer is calculated at 323.

[0065] The energy transfer can include potential energy transferred to the movement and momentum of an object that falls from a height within the scene. Alternatively, the energytransfer can include heat dissipation from a hot object to the user or an object in proximity (convective heat loss) or in contact (conductive heat loss) with the object. Once the propagation is complete at 325, the energy transfers are known, and the energy state of each object can be updated at scene update 330. The properties of an object are updated at 331, such as if an object fell from a height, the object will have lost potential energy. Interactions are adjusted for at 332, where interactions between the user and objects or objects with other objects are accounted for at this stage.

[0066] The scene is updated at 335 based on the user input and the change in energy states of the virtual objects within the scene. Feedback is rendered at 340, which energy mapped to devices at 340 and fallback mechanisms are applied at 342. Feedback is rendered in the form of haptic feedback to a user through a haptic device. The mapping of energy to devices maps the type of energy interactions / transfers for the one or more virtual objects based on the object properties and the energy transformations. The energy is mapped to the appropriate feedback mechanism of the haptic device based on the device capabilities. If the haptic device does not have feedback capabilities corresponding to a particular energy transformation, a fallback mechanism may be employed, where a certain energy transformation (e.g., thermal transformation) is mapped to a vibrational haptic feedback response at the haptic device.Feedback is rendered at 345 to the haptic device, and the process awaits the next interaction in the idle / wait state 350. During the idle / wait state 350, inactivity is monitored at 351 and trigger detection is awaited at 352. Upon trigger detection (e.g., of user input), new input detection 360 is conveyed to the interaction detection 310 for processing as described above.

[0067] The scene dynamics of certain example embodiments are governed by equations that describe the global behavior of the virtual environment, ensuring energy conservation, dynamic updates, and realistic propagation to connected devices thereby modeling the first law of thermodynamics. In doing so, total energy conservation can be represented by the following equation:Etotal= Ethermal+ Ekinetic+ Evibration+ Epotential+ ...Where Etotai is the total energy of the virtual scene, Ethermal is the thermal energy of the objects in the scene, Ekineticis the kinetic energy of the objects in the scene, Evibration is the vibrationalenergy of objects in the scene, and Epotentialis the potential energy of the virtual objects in the scene.

[0068] Equation (1) ensures all energy transformations within the virtual scene are conserved and reflected through the model. This serves as the foundation of the modeling framework ensuring the system remains physically realistic. All subsequent energy transformations feed back into the original Equation (1).

[0069] The dynamic energy of a scene provides the ambient energy or thermal state of the entire scene by averaging object-specific energy values. The dynamic scene energy acts as a reference point for object-specific updates and feedback rendering ensuring that the environment evolves consistently. Equation (2) below represents dynamic scene energy:scene, dynamic J ^ fm -c ■) where Escene, dynamic is the ambient energy of the scene, with Ei representing the energy of an object, m; representing the mass, and Ci representing the thermal constant.

[0070] Energy propagation provides intelligent mediation of a feedback signal from a source to a point of contact within the scene. Energy propagation models energy transfer (e.g., heat, vibration) across intermediary objects or mediums within the scene as shown using Equation (3). This provides the energy that propagates through the system, feeding into both object updates and feedback rendering through haptic devices.Epropagated Esource’ fit (1 TG -^ ' ‘ 0) where Epropagatedis the propagated energy, Esourceis a source of the propagated energy as initial energy introduced such as from collision or user input, and the product of each object within the environment, where TL reflects the transmission loss for object i, TG reflects the transmission gained for object i with a being the attenuation coefficient and d, being the distance traveled in medium i. The attenuation coefficient can be represented by:a= +^dir’cos 9(4) where p is density of the medium, Y is Young’s modulus for stiffness, Z is impedance, y is a damping coefficient, βdiris the directional attenuation constant for the medium, and 0 is the angle of energy propagation relative to the medium’s primary axis. The medium dependent attenuationquantifies energy dissipation based on the properties of the medium (e.g., stiffness and density) and is used to ensure realistic energy decay as it moves through the scene.

[0071] In addition to maintaining appropriate virtual scene dynamics, certain embodiments determine how individual objects within the scene interact with one another and update their energy states over time. The energy state for each object is updated based on interactions with neighboring objects, such as through heat conduction, collision, vibrational forces, etc. Equation 5 below reflects the energy state E, for object i at a point in time t.Ei(t + 1) = £)(t) + -i- S; (kij ■ Atj■ - E^t)}) ■ e-^ (5)where E_i(t) is Energy of object i at time t, k_ij is the transfer coefficient between objects i and j, A_ij is the contact area between objects, C_i, m_i is the energy capacity and mass of object i, λ_i is the decay constant for dissipation.

[0072] Energy transformation rules are employed to model how energy changes form, such as potential to kinetic, kinetic to thermal, etc. during interactions such as collisions or falls. The energy transformation rules feed into the total energy conservation equation ensuring all transformations are tracked.E thermal ~ ^1 ' Eyinetic > Ey[net[c— Tfl • g • h (6) where Ethermai is the thermal energy, Ekineticis the kinetic energy, and g is an efficiency factor for converting kinetic energy to thermal energy of the object. Epotentiai is the potential energy, with in being the mass, g being gravitational constant, and h being the height of the object.

[0073] Feedback rendering is used to calculate how the propagated energy is rendered into haptic feedback by connected devices.^device, input ^feedback Y\device (7) where E_feedback is total feedback energy transmitted to the device and η_device is the efficiency of the device in converting energy into haptic feedback. Each device renders specific types of feedback. For n-types of feedback:^■device, input=Xn E device, n (8) where E_device,n is energy allocated for rendering feedback type n (thermal, vibrational, etc.). This is integrated into the propagation equation where it receives input and determines how much energy is available for rendering.

[0074] Device-specific feedback models render energy as compatible with device capabilities. For vibrotactile feedback, Equation (9) represents how energy is rendered as vibrations optimized for resonance frequency.^vibration, output ^vib J ^vib &f(9) where E_vibration, output is the output energy as vibration, A_vib is actuator amplitude, f is input frequency, f_res is resonance frequency, Δf is bandwidth around the resonance frequency, and η_vib is the efficiency factor for vibrational rendering.

[0075] Thermal feedback converts energy into heat transfer based on contact area and temperature gradient.Qthermcil=Kthermai ' AT ' Acontact ' thermal (10) where K_thermal is the thermal conductivity of the actuator, ΔT is the temperature gradient (T_feedback - T_device), A_contact is the contact area between the actuator and skin, and η_thermal is the efficiency factor for thermal feedback. Material properties can be dynamically updated with Equation (11):kthermal=^0 ’ (1 + / ? ’ (J—^o)) (H) where K_thermal is the thermal conductivity at temperature T, β is the temperature coefficient, and T_0 is the reference temperature.

[0076] In addition to vibrational feedback, thermal feedback, and kinetic feedback, at least some embodiments provided herein can implement pneumatic feedback where appropriate.Certain embodiments modulate pressure output proportional to the available feedback energy. Equation (12) illustrates how this pneumatic output can be calculated.D>. D. E feedback, pneumaticoutput ~ ' / pneumatic max Emaxwhere P_output is output pressure, P_max is maximum pressure capability of the pneumatic system, E_feedback, pneumatic is the energy available for pneumatic feedback, E_max is the energy required to achieve P_max, and η_pneumatic is the energy factor for pneumatic feedback.

[0077] Energy in the virtual environment is treated as conserved since transformations (e.g., potential to kinetic) occur within the scene, and energy dissipation is modelled as heat orvibration but remains accounted for in the virtual system. A user can introduce energy to the system, such as through squeezing, pushing, or impacting objects. These inputs dynamically propagate through the scene to help produce haptic feedback.

[0078] Kinetic energy propagates through collisions or movement. Potential energy converts to kinetic when an object falls or is displaced. Thermal energy is generated through friction, collisions, or transformations. Chemical energy is generated through forming or breaking of bonds within molecules and through phase changes. Electrical energy models electro-tactile feedback. Vibrational energy is generated through impacts, motor driven devices, or oscillations. The system of at least some example embodiments dynamically updates energy propagation and feedback based on interaction density, scene complexity, and device capabilities.

[0079] To address latency issues in high frequency adaptive time-step mechanism, certain embodiments adapt dynamically based on user interaction or computational load. Equation 13 represents this as:ZAC = - • K (13)AC scenev 7where Δt is a change in time, ΔE_interaction is energy change due to user interaction, ΔE_scene is total energy change in the scene, and k is a scaling constant for computational efficiency.Additionally, latency modeling can incorporate the following for feedback delivery:^feedback=~ + ^processing (14) where d is the propagation distance, c is the transmission speed, and t_processing is the computational delay.

[0080] Certain embodiments described herein rely on device-specific modeling, where each haptic device (e.g., vibrotactile actuators, thermal systems, pneumatic systems) has unique physical properties and rendering constraints. The model dynamically adapts energy feedback to match the device’s capabilities. At least some embodiments ensure that the energy propagated to a device aligns with the type of feedback it can generate (e.g. thermal energy to a Peltier device, vibrational energy to a vibrotactile actuator, etc.). Models of certain example embodiments can employ device-specific parameters when mapping energy transformations to a feedback device.These device-specific parameters can include an efficiency factor which includes a fraction of input energy effectively rendered as output feedback and a resonance frequency which is the optimal frequency at which the device operates efficiently. Each device renders specific types of feedback. For n-types of feedback:Edevice, input ~ device, n 0^) Where Edevice,nis the energy allocated for rendering feedback type n (thermal, vibrational, etc.).

[0081] A total feedback energy allocation can be spread across devices as shown by Equation (16):^device, total ~ ^device E device, input (16) According to certain embodiments described herein, a device can receive a proportion of energy based on its priority for rendering specific feedback and its efficiency in converting input energy.

[0082] Haptic feedback devices operating near their resonance frequency or optimal range receive prioritized energy allocation as shown in Equation 17:V device ‘ ^device ^device y ~ „ 07)Lj Vj ' Kj Where W_device is a weighting factor of the device, and R_device is the relevance of the device for rendering specific feedback which can be user defined or context dependent.

[0083] The general process for rendering multi-device feedback in a manner similar to that illustrated above with respect to Figure 3 can include an initializing operation, distributed feedback energy determination, rendering feedback to the devices, and adjusting feedback in real-time. Initialization of devices includes loading of device parameters (η_device, f_res, K_thermal, P_max, etc.). Energy constraints and capabilities are retrieved from each haptic output device.

[0084] To distribute feedback energy, the device input energy is calculated for each device based on its efficiency and feedback requirements. The energy can be allocated proportional to device relevance. To render feedback, specific feedback parameters may be determined. For vibrotactile devices, a vibration output energy is computed and modulated based on frequency. For thermal devices, the heat transfer rate may be computed and heat transfer modulated. For pneumatic devices, the pressure output can be computed and the pressure modulated. A user response can be monitored for contact quality, proximity, etc. and feedback energy candynamically be adjusted based on the user interaction and device efficiency to adjust feedback in real-time.

[0085] The model described herein according to certain example embodiments, recalculates each object’s properties in a given unit of time (calculated depending on computational resources allocated for modeling thermal interaction) based on its own properties, interacting with other objects, and the scene. This may be of particular relevance for thermal feedback in a virtual scene. Certain example embodiments therefore allow for adaptation of entropy in any virtual scene containing more than one object, a heat source and / or a heat sink. Updating object temperature within unit time “t” depends on the computational resources allocated to the rendering model as well as the frequency and resolution of the output (haptic) device.

[0086] According to certain example embodiments, thermal feedback signals are converted into alternative haptic outputs when thermal feedback devices are unavailable. Thermal signals may be mapped to vibrotactile signals proportional to the thermal intensity. Conversion parameters can include scaling factors based on user interaction and environmental entropy changes. According to certain example embodiments, such a fallback mechanism can therefore dynamically adjust based on the capabilities of the connected haptic devices.

[0087] An example embodiment of a virtual reality interaction within a virtual scene includes a user picking up a metal rod and using it to strike a bell. According to the example, the user can grab the metal rod, applying a force (Fuser) of 50 Newtons as the external energy input. The energy calculation can be represented by:E_kinetic = ½ · m · v² (18)If the rod object has a mass of 2 kilograms and the user swings the rod at 2 meters per second, the kinetic energy introduced is represented by:XEkinetic = '2‘ (2)2=4 4E kinetic ~ 2 ’ 2 ’ (2) — 4 J (19)

[0088] The interaction within the virtual scene of striking the bell results in some energy being transformed into vibration and sound. The energy transmitted to the bell, using Equation 3:T1‘I-‘rod-bellE transmitted ~ E kineticEGrod-beiiAssume TL_rod-bell = 0.2 and TL_rod-bell = 0.9 :E_transmitted = 3.11 J The vibrational energy is calculated as:E_vibration = η_vibration · E_transmittedWith η_vibration = 0.8:E_vibration = 0.8 · 3.11 = 2.49 J

[0089] The above equations are used to establish an energy to provide as feedback to the user through the haptic device. The vibration of the bell can be propagated to the user’s hand as represented by the following:Cj-, — Z7 • >^feedback ~ ^vibrationbbellAssume S_contact = 0.01m² and S_bell = 0.1m²:0.01E_feedback = 2.49 · (0.01 / 0.1) = 0.249 Jwhere S_contact is the contact surface area and S_bell is the total surface area of the interacting object (e.g., the bell).

[0090] The rod can then be placed on a vibrating platform which oscillates at / =50Hz with amplitude zl=0.02meters. The energy introduced to the platform is represented by the equation:1 7.7Eplatform = - • m ' at ' A2where <jj=27t 'is the angular frequency. For a mass of 2 kilograms:ω = 2π · 50 = 314.16 rad / sEplatform = | ’ 2 • (314.16)2• (0.02)2= 39.47 J

[0091] Energy from the platform can be dissipated through a rubber pad on which the platform rests. For energy loss through the medium of a rubber pad with ρ = 1200kg / m³, Y=0.01 Pa, Z= 50Ω and γ = 0.05:0.05 • 1200= 10.950.01 • 50Attenuation over d=0. Im:E_propagated = E_platform · e^(-α·d) = 19.73 · e^(-10.95·0.1) = 6.38 JFeedback in the form of vibrational energy is relayed to the rod and then the user.

[0092] Interaction with thermal energy can be provided through feedback to a user through a haptic device as described above. One such example is a rod that is heated to a temperature of 500K (Tfmai = 500K). The thermal energy can be represented as:E_thermal = C_p · m · ΔTAssuming specific heat capacity Cp = 500J / kg:K, ΔT = 500 - 300 = 200K, m = 2 kgE_thermal = 500 · 2 · 200 = 200,000 J or 200 KJEnergy transfer to the user employs the contact transfer rate through metal (k=50):Efeedback ^contact ’ tFor Scontact = 0.01m2, t = Is, the feedback energy to the user is:E_feedback = 50 · 200 · 0.01 · 1 = 100 JThis may be provided as feedback to the user through a haptic device using a Peltier device, for example. If thermal feedback is not available with a particular haptic device, the energy may be provided as feedback to the user through a vibrational response, for example.

[0093] Although the updating of properties of virtual objects within the scene can occur in the background even without the user carrying out any interaction within the virtual environment, a simulated process can be slowed to ensure time is provided to the user to engage with the objects, such as with an object’s thermal decay. As another example, if the user starts engaging with the scene, the environment or object within, the process of entropy (heat transfer between objects) starts to occur faster. According to certain example embodiments, the thermal entropy in the scene can be updated at a faster rate (perhaps hundreds of times per second) when the users are actively engaging with the environment or object(s), and the rate can slow down when the user is not engaging actively. Similarly, the vibration described above with respect to the bell being rung may decay over time. The concepts can be described as time dilation, meaning, time slows down when the user is inactive and speeds up when the user is active. This can also be employed for objects losing potential energy (e.g., falling or descending from a height, dissipating vibrational energy, etc.). The rate of flow of time can be directly proportional to entropy (rate by which thermal energy is dissipated between an object and the scene in general). This process can continue until maximum entropy is reached (meaning when average scene temperature and all object temperatures reach a thermal equilibrium) or when no additional energy translations are occurring, and object properties remain substantially constant. During this process the temperature of the objects within the scene are relayed to the user when they touch the object or come close to it. Depending on the object temperature, conductivity, and the background scene temperature (e.g., air, water, space etc.) thermal feedback can be created and relayed to the user through conduction, convection and radiation.

[0094] According to certain example embodiments, if the scene contains a hot cup of coffee, it may start losing its thermal energy to its surroundings (air, cup table etc.) even if the user doesn’t interact with the cup. But once the user comes close to it or touches the cup, the latesttemperature of the cup and the coffee can be rendered via the user device. According to certain example embodiments, heat is radiated once the user comes closer to the hot object. Similarly, heat from the user can move towards a cooler object (such as ice etc.) once the user comes close to it or touches it directly. These values may be calculated and updated frequently, such as hundreds of times per second, and relayed to the user (via a thermal feedback device or other haptic feedback device) if the user is interacting with objects frequently or continuously. As another example, the updates may occur less frequently if the user is not actively interacting with the object or scene.

[0095] Certain example embodiments may therefore incorporate the concept of time dilation (rate of change of thermal decay slowing down). If the user is actively interacting with objects in the scene, then all the objects in the scene may lose and / or gain energy from their surrounding depending on how active the user is. Whereas, if the user disengages from the scene and virtual environment, the energy decay (such as thermal energy decay) may continue but may become slower. According to certain embodiments, multiple rates and / or modes of thermal decay may be implemented, such as slow, medium and fast, based on an engagement level of a user falling into corresponding thresholds. According to certain embodiments, a rate of energy decay can be dynamic and proportional to the user’s engagement level. In any event, certain example embodiments provide a non-linear time progression based on one or more user interactions with the virtual environment, a determined thermal energy transfer, and an absolute temperature. Heat propagation and hence increase in entropy within the environment may speed up or slow down depending on the engagement of the user. Entropy stabilization occurs once all objects (heat sources and sinks) reach average scene temperature or motion of the objects and their interaction does not yield any further energy translation (potential to kinetic; kinetic to thermal, etc.).

[0096] According to certain embodiments, once the user comes close or in contact with an object and heat or feeling of coolness needs to be relayed, the original properties of the object (such as, conductivity, emissivity, radiance, mass, heat capacity, surface area, etc.) are utilized to render the thermal perception of the object to the user, optionally employing the attached device parameters. This means that two objects with the same temperature may create different perceptual output for the user. For example, while baking a cake, you may open an oven and touch the surface of the cake, without getting burnt, while if you touch the metal container in the same oven at the same temperature, a user can burn their hand, because of the properties of theobject dictate how heat flows from it to the surroundings. Similarly, in the virtual environment, according to certain example embodiments, these properties are tracked and updated, so that once the uses engage with an object, the properties can be used to create a realistic and immersive thermal perception via a user device. Certain example embodiments normalize the thermal feedback across user devices, so that the experience is consistent irrespective of the device, its technology, its maximum and minimum temperature outputs, or the rate of change (time needed) to adjust the temperature within the device.

[0097] Figures 4 and 5 are flowcharts of operations performed by apparatus 200 according to an example embodiment. As shown by operation 410 in Figure 4, apparatus 200 may include means, such as the processor 220, user interface 222, communication interface 224, memory device 226, or the like, for identifying one or more virtual objects within a virtual scene. Object properties for the one or more virtual objects within the virtual scene are determined at 420, which may be performed by means, such as the processor 220, of the apparatus 200 as described above. Energy transformations are calculated for the one or more virtual objects at 430, which may be performed, for example, by means, such as the processor 220, of the apparatus 200. Energy interactions are encoded for the one or more virtual objects at 440 based on the object properties and energy transformations. These interactions may be encoded, such as by means, such as the processor 220, and stored by means, such as the memory device 226. The energy interactions for the one or more virtual objects are mapped to energy output for one or more haptic devices. The output may be performed, for example, by means, such as the haptic feedback device 30, of an example embodiment.

[0098] As shown in operation 510 of Figure 5, energy interactions with one or more virtual objects within a virtual scene are encoded based on object properties of the one or more virtual objects, interactions with the one or more virtual objects, and energy transformations associated with the one or more virtual objects. This may be performed by apparatus 200 which may include means, such as processor 220, for encoding the energy interactions and means, such as the memory device 226, to store the interactions, for example. Energy is propagated across the one or more virtual objects at 520, such as by means, such as the processor 220, of the apparatus 200. Energy states of the one or more virtual objects are updated at 530, which may be performed by means, such as the processor 220, and stored by means, such as the memory device 226, of the apparatus 200, for example. The energy interactions of the one or more virtual objectsare mapped to energy outputs for the one or more haptic devices based at least in part on the energy states of the one or more virtual objects. The energy outputs may be realized by means, such as the haptic feedback device 30, for example.

[0099] A system is therefore provided according to certain example embodiments for modeling objects in a virtual environment, and more particularly, to modeling object behaviors and providing haptic signal mediation through energy propagation in virtual environments. Certain example embodiments model a virtual environment defined as a closed system, wherein energy transformations, including thermal, kinetic, and potential, are translated, converted and updated, following the laws of energy conservation and entropy within a closed system.. A mathematical framework is provided for calculating object-specific property and energy changes based on interactions (e.g., user-object and object-object), dynamic scene properties, and feedback, which can be rendered via a haptic feedback system, including but not limited to vibrotactile actuators, thermal systems, force feedback systems and / or pneumatic systems like. Certain example embodiments provide energy conserving haptic feedback based on user interaction to provide improved realistic and immersive experiences to users of virtual reality systems.

[0100] Conserving energy within a virtual environment as described herein includes mediation and propagation of energy as object-to-object and / or user-to-object interactions occur. For example, if a ball falls off a shelf from a rack onto a table below, the potential energy of the ball is translated to kinetic energy, then sound (e.g., sound waves of the ball bouncing on the table), and then kinetic energy felt through the pressure wave of the impact of the ball on the table, which may translate into motion of other objects placed on the surface of the table. These translations would occur according to the law of conservation of energy and increase in entropy. The efficiency of energy conversion is not 100% outside of a vacuum, such that after a few conversion cycles of energy, the resistive forces described within the virtual environment (such as within a medium of interaction including air, water, etc.) including gravity, air / fluid resistance, friction, momentum, etc. will ensure that the energy is saturated within the system and maximum entropy is reached within the virtual environment.

[0101] The propagation or mediation of energy is similar irrespective of the type of material of the objects interacting with each other or which medium the interaction is taking place in. If a rubber ball falls off of a shelf onto a metal table versus a steel ball falling off of a shelf onto themetal table, the sequence of translations of energy would be the same but because the objects have different starting properties, the mediation and propagation (and hence the output energy envelope) would be very different. The same principle can be applied to the medium of interaction if the virtual scene medium is water, air, or in space, the interaction properties may change but the way energy is translated will remain the same.

[0102] A system can utilize certain example embodiments disclosed here to implement, test, and / or optimize a framework’s ability to calculate and conserve energy, within a closed or open virtual system. Certain example embodiments can develop, optimize, and / or integrate haptic feedback devices, covering multiple technologies (thermal, vibrotactile, pneumatic) using energy conservation within a scene where object or environment properties change with the passage of time or through direct or indirect user interaction, ensuring that any localized energy loss in one object is matched by a corresponding energy increase in other object(s) within the closed system. Certain example embodiments therefore highlight the procedural aspects of dynamic interaction by calculating feedback, modulation, or energy conservation in a virtual environment, and defines or includes real-time updates to a virtual scene, object or interaction in a virtual environment as time progresses with or without user interaction-dependent adjustments, therefore simulating energy conservation in the virtual environment. Output and / or renderings can be tailored to specific haptic technologies, such as Peltier or other thermal feedback actuators in the presence or absence of vibrotactile, kinesthetic or pneumatic feedback renders. The output and / or renderings can utilize the concept of energy conservation in a closed or open virtual environment with accurate modeling of energy transfer and redistribution within a closed system to ensure modeling and delivery of realistic interactions.

[0103] As described above, Figures 4 and 5 illustrate flowcharts of an apparatus 200, method, and computer program product according to certain example embodiments of the disclosure. It will be understood that each block of the flowchart, and combinations of blocks in the flowchart, may be implemented by various means, such as hardware, firmware, processor, circuitry, and / or other devices associated with execution of software including one or more computer program instructions. For example, one or more of the procedures described above may be embodied by computer program instructions. In this regard, the computer program instructions which embody the procedures described above may be stored by a memory device 226 of an apparatus 200 employing an embodiment of the present disclosure and executed by aprocessor 220 of the apparatus 200. As will be appreciated, any such computer program instructions may be loaded onto a computer or other programmable apparatus (e.g., hardware) to produce a machine, such that the resulting computer or other programmable apparatus implements the functions specified in the flowchart blocks. These computer program instructions may also be stored in a computer-readable memory that may direct a computer or other programmable apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture, the execution of which implements the function specified in the flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operations to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide operations for implementing the functions specified in the flowchart blocks.

[0104] Accordingly, blocks of the flowchart support combinations of means for performing the specified functions and combinations of operations for performing the specified functions for performing the specified functions. It will also be understood that one or more blocks of the flowchart, and combinations of blocks in the flowchart, may be implemented by special purpose hardware-based computer systems which perform the specified functions, or combinations of special purpose hardware and computer instructions.

[0105] In some embodiments, certain ones of the operations above may be modified or further amplified. Furthermore, in some embodiments, additional optional operations may be included. Modifications, additions, or amplifications to the operations above may be performed in any order and in any combination.

[0106] Many modifications and other embodiments of the disclosures set forth herein will come to mind to one skilled in the art to which these disclosures pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the disclosures are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, although the foregoing descriptions and the associated drawings describe certain example embodiments in the context of certain example combinations of elements and / or functions, it should be appreciated that different combinations of elementsand / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and / or functions than those explicitly described above are also contemplated as may be set forth in some of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

THAT WHICH IS CLAIMEDCLAIMS1. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least:identify one or more virtual objects in a virtual scene;determine object properties for the one or more virtual objects;calculate, for the one or more virtual objects, energy transformations;encode energy interactions for the one or more virtual objects based on the object properties and the energy transformations; andmap the energy interactions for the one or more virtual objects to energy output for one or more haptic devices.

2. The apparatus of claim 1, wherein causing the apparatus to determine object properties for the one or more virtual objects comprises causing the apparatus to:assign an energy signature to the one or more virtual objects, wherein the energy signature is associated with a position of the one or more virtual objects within the virtual scene, a proximity of the one or more virtual objects to one or more other virtual objects within the virtual scene, and an interaction between the one or more virtual objects and the one or more other virtual objects within the virtual scene.

3. The apparatus of claim 1 or 2, wherein the energy interactions comprise one or more of kinetic energy, thermal energy, vibrational energy, potential energy, chemical energy or electrical energy.

4. The apparatus of claim 3, wherein causing the apparatus to map the energy interactions for the one or more virtual objects to energy output for one or more haptic devices comprises causing the apparatus to:determine supported functions of the one or more haptic devices; andmap the energy interactions for the one or more virtual objects to the supported functions.

5. The apparatus of claim 4, wherein causing the apparatus to map the energy interactions for the one or more virtual objects to energy output for one or more haptic devices comprises causing the apparatus to:convert the energy interactions from at least one of kinetic energy, thermal energy, vibrational energy, potential energy, chemical energy, or electrical energy to at least one other of kinetic energy, thermal energy, vibrational energy, potential energy, chemical energy or electrical energy based on the supported functions.

6. The apparatus of claim 3, wherein causing the apparatus to calculate, for the one or more virtual objects, energy transformations comprises causing the apparatus to:calculate energy transformations for the one or more virtual objects based, at least in part, on object properties of the one or more virtual objects, wherein the object properties comprise one or more of density, thermal conductivity, coefficient of restitution, or a spring constant.

7. The apparatus of any of claims 1 to 6, wherein the apparatus is further caused to:calibrate the energy output to an energy output range for the one or more haptic devices based on identified capabilities of the one or more haptic devices.

8. The apparatus of claim 7, wherein causing the apparatus to calibrate the energy output to an energy output range for the one or more haptic devices based on identified capabilities of the one or more haptic devices comprises causing the apparatus to:identify the identified capabilities of the one or more haptic devices based on an initialization of at least one of the virtual scene or the one or more haptic devices.

9. The apparatus of any of claims 1 to 8, wherein causing the apparatus to determine the object properties for the one or more virtual objects comprises causing the apparatus to:determine time-dependent object properties; andattenuate the time-dependent object properties over time.

10. The apparatus of claim 9, wherein causing the apparatus to attenuate the time-dependent object properties over time comprises causing the apparatus to:determine that a user is not interacting with the one or more virtual objects; and reduce a frequency at which the time-dependent object properties are attenuated over time in response to determining that the user is not interacting with the one or more virtual objects.

11. The apparatus of claim 10, wherein causing the apparatus to attenuate the time-dependent object properties over time comprises causing the apparatus to:determine that a user is interacting with the one or more virtual objects; andincrease a frequency at which the time-dependent object properties are attenuated over time in response to determining that the user is interacting with the one or more virtual objects.

12. The apparatus of any of claims 1 to 11, wherein the apparatus is further caused to:synchronize the virtual scene based on a rendering frequency of the one or more haptic devices.

13. A method comprising:identifying one or more virtual objects in a virtual scene;determining object properties for the one or more virtual objects;calculating, for the one or more virtual objects, energy transformations;encoding energy interactions for the one or more virtual objects based on the object properties and the energy transformations; andmapping the energy interactions for the one or more virtual objects to energy output for one or more haptic devices.

14. The method of claim 13, wherein determining object properties for the one or more virtual objects comprises:assigning an energy signature to the one or more virtual objects, wherein the energy signature is associated with a position of the one or more virtual objects within the virtual scene, a proximity of the one or more virtual objects to one or more other virtual objects within the virtual scene, and an interaction between the one or more virtual objects and the one or more other virtual objects within the virtual scene.

15. A method for rendering haptic feedback in a virtual environment comprising: encoding energy interactions with one or more virtual objects within a virtual scene based on object properties of the one or more virtual objects, interactions with the one or more virtual objects, and energy transformations associated with the one or more virtual objects;propagating energy across the one or more virtual objects;updating energy states of the one or more virtual objects; andmapping the energy interactions of the one or more virtual objects to energy outputs for one or more haptic devices based at least in part on the energy states of the one or more virtual objects.

16. The method of claim 15, wherein propagating energy across the one or more virtual objects comprises:propagating energy from a first object of the one or more virtual objects to a second object of the one or more virtual objects, wherein the energy from the first object to the second object is propagated at least one of directly between the first object and the second object or indirectly through an element of the virtual scene.

17. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least:encode energy interactions with one or more virtual objects within a virtual scene based on object properties of the one or more virtual objects, interactions with the one or more virtual objects, and energy transformations associated with the one or more virtual objects;propagate energy across the one or more virtual objects;update energy states of the one or more virtual objects; andmap the energy interactions of the one or more virtual objects to energy outputs for one or more haptic devices based at least in part on the energy states of the one or more virtual objects.

18. The apparatus of claim 17, wherein causing the apparatus to propagate energy across the one or more virtual objects comprises causing the apparatus to:propagate energy from a first object of the one or more virtual objects to a second object of the one or more virtual objects, wherein the energy from the first object to the second object is propagated at least one of directly between the first object and the second object or indirectly through an element of the virtual scene.

19. The apparatus of claim 17 or 18, wherein updating the energy states of the one or more virtual objects is performed at a frequency proportional to user interaction with the one or more virtual objects.

20. The apparatus of any of claims 17 to 19, wherein updating the energy states of the one or more virtual objects is performed at a frequency based on an output frequency of the one or more haptic devices.

21. The apparatus of any of claims 17 to 20, wherein mapping the energy interactions of the one or more virtual objects to energy outputs for the one or more haptic devices is performed based on supported energy outputs for the one or more haptic devices.

22. The apparatus of claim 21, wherein the energy interactions comprise one or more of kinetic energy, thermal energy, vibrational energy, potential energy, or electrical energy.

23. The apparatus of claim 22, wherein causing the apparatus to map the energy interactions of the one or more virtual objects to the energy outputs for one or more haptic devices comprises causing the apparatus to:convert the energy interactions from at least one of kinetic energy, thermal energy, vibrational energy, potential energy, or electrical energy to at least one other of kinetic energy, thermal energy, vibrational energy, potential energy, chemical energy, or electrical energy based on supported functions of the one or more haptic devices.

24. The apparatus of any of claims 17 to 23, wherein the apparatus is further caused to:calibrate the energy outputs to energy output ranges for the one or more haptic devices based on identified capabilities of the one or more haptic devices.

25. The apparatus of any of claims 17 to 24, wherein the apparatus is further caused to: determine time-dependent object properties; andattenuate the time-dependent object properties over time.

26. The apparatus of claim 25, wherein causing the apparatus to attenuate the time-dependent object properties over time comprises causing the apparatus to:determine that a user is not interacting with the one or more virtual objects; and reduce a frequency at which the time-dependent object properties are attenuated over time in response to determining that the user is not interacting with the one or more virtual objects.

27. The apparatus of claim 26, wherein causing the apparatus to attenuate the time-dependent object properties over time comprises causing the apparatus to:determine that a user is interacting with the one or more virtual objects; andincrease a frequency at which the time-dependent object properties are attenuated over time in response to determining that the user is interacting with the one or more virtual objects.

28. A computer program comprising instructions for performing at least the following:identifying one or more virtual objects in a virtual scene;determining object properties for the one or more virtual objects;calculating, for the one or more virtual objects, energy transformations;encoding energy interactions for the one or more virtual objects based on the object properties and the energy transformations; andmapping the energy interactions for the one or more virtual objects to energy output for one or more haptic devices.

29. A computer program comprising instructions for performing at least the following:encoding energy interactions with one or more virtual objects within a virtual scene based on object properties of the one or more virtual objects, interactions with the one or more virtual objects, and energy transformations associated with the one or more virtual objects;propagating energy across the one or more virtual objects;updating energy states of the one or more virtual objects; andmapping the energy interactions of the one or more virtual objects to energy outputs for one or more haptic devices based at least in part on the energy states of the one or more virtual objects.

30. An apparatus comprising:means for identifying one or more virtual objects in a virtual scene;means for determining object properties for the one or more virtual objects;means for calculating, for the one or more virtual objects, energy transformations; means for encoding energy interactions for the one or more virtual objects based on the object properties and the energy transformations; andmeans for mapping the energy interactions for the one or more virtual objects to energy output for one or more haptic devices.

31. An apparatus for rendering haptic feedback in a virtual environment comprising:means for encoding energy interactions with one or more virtual objects within a virtual scene based on object properties of the one or more virtual objects, interactions with the one or more virtual objects, and energy transformations associated with the one or more virtual objects;means for propagating energy across the one or more virtual objects;means for updating energy states of the one or more virtual objects; andmeans for mapping the energy interactions of the one or more virtual objects to energy outputs for one or more haptic devices based at least in part on the energy states of the one or more virtual objects.

Images