A video game processing method, and a device and system for performing the same

Abstract

A video game processing method, and a device and system for performing the same A video game processing method, comprising the steps of: outputting a video game for display to a user; obtaining orientation data from a first sensor, the orientation data indicating an orientation of the user's head relative to a datum orientation; determining, based on the orientation data, whether the orientation of the user's head falls outside of a predetermined range of orientations at which a first region of the video game's virtual environment is visible to the user; and if so, modifying the outputting of at least the first region of the video game's virtual environment.

Description

A VIDEO GAME PROCESSING METHOD, AND A DEVICE AND SYSTEM FOR PERFORMING THE SAME

[0001] The present invention relates to the field of video game processing methods, and devices and systems for performing the same.Background

[0002] In the field of video game processing, the term “popping” refers to sudden visual changes occurring on a screen displaying a video game, such sudden visual changes resulting from sudden changes to the level of detail of a given virtual element (an in-game character or object, for example). Similarly, the term “pop-in” refers to sudden visual changes resulting from sudden (dis)appearances of the given virtual element itself. Such sudden visual changes may be jarring / discomforting for a user to view, especially if they occur frequently and / or affect virtual elements which take up increasingly larger amounts of screen space. Additionally, depending on the frequency, such sudden visual changes may even dangerous to certain users, especially those with epilepsy, as such visual change may result in the user experiencing seizures.

[0003] Meanwhile, users playing video games typically look away from the screen occasionally (to talk to other people, to interact with their smartphone, to look outside a window, or the like). However, doing so while the video game is ongoing may adversely affect the user’s gameplay experience (the user’s virtual character may be injured / killed, the user’s virtual vehicle may crash, or the like).

[0004] Embodiments of the present description seek to alleviate or mitigate these issues.

[0005] In a first aspect, there is provided a video game processing method, comprising the steps of: outputting a video game for display to a user; obtaining orientation data from a first sensor, the orientation data indicating an orientation of the user’s head relative to a datum orientation; determining, based on the orientation data, whether the orientation of the user’s head falls outside of a predetermined range of orientations at which a first region of the video game’s virtual environment is visible to the user; and if so, modifying the outputting of at least the first region of the video game’s virtual environment.

[0006] Optionally, the modifying step comprises initiating a game event within the at least first region of the video game’s virtual environment.

[0007] Optionally, initiating the game event comprises one or more of: pausing the video game; instantiating a virtual element within the first region of the video game’s virtual environment; modifying a state or attribute of an instantiated virtual element that is located within the first region of the video game’s virtual environment; and outputting an audio signal and / or haptic signal corresponding to an instantiated virtual element that is located within the first region of the video game’s virtual environment.

[0008] Optionally, the outputting step comprises outputting the at least first region of the video game’s virtual environment to a standalone display screen for display to the user.

[0009] Optionally, the standalone display screen is one of: a television screen; a computer monitor; a laptop screen; a tablet computer screen; a smartphone screen; and a screen of a portable video game console.

[0010] Optionally, the first sensor comprises an inertial measurement unit, IMU, wearable on the user’s head and configured to measure the orientation of the user’s head.

[0011] Optionally, the IMU is coupled to a head-wearable audio system.

[0012] Optionally, the head-wearable audio system is one of: a pair of headphones; a pair of bone conduction headphones; and a pair of earphones.

[0013] Optionally, the first sensor comprises a light sensing element wearable on the user’s head and configured to sense light incident thereon and generate orientation data based on one or more illumination characteristics of the sensed light; and the determining step comprises determining whether the orientation data fails to satisfy any of one or more illumination criteria, the modifying step being carried out if so.

[0014] Optionally, the light sensing element is coupled to a head-wearable audio system.

[0015] Optionally, the light sensing element comprises one or more of: a photodiode; and a camera.

[0016] Optionally, the one or more illumination criteria comprise one or more of: a light colour; a light intensity; a change in light colour; a change in light intensity; and a frequency range or wavelength range of the light.

[0017] Optionally, the method comprises the steps of: obtaining rendered image frames of the video game; inserting a calibration frame after every N rendered image frames of the video game, wherein N ≧ 1, wherein each inserted calibration frame comprises predetermined image data for satisfying one or more of the illumination criteria; and outputting the video game having the inserted calibration frames.

[0018] Optionally, the outputting step comprises: obtaining the orientation data from a second sensor; determining, based on the orientation data from the second sensor, which region of the video game’s virtual environment is to be output to one or more screens of a head-mounted display, HMD, system; and outputting the determined region of video game’s virtual environment to the one or more screens of the HMD system for display to the user.

[0019] Optionally, the first sensor is one of: an inertial measurement unit, IMU, of the HMD system, the IMU being configured to measure the orientation of the user’s head; and a standalone camera configured to capture images of the HMD system when worn by the user.

[0020] Optionally, the second sensor is the other of: the IMU; and the standalone camera. Alternatively, the second sensor is the same as the first sensor.

[0021] Optionally, the method comprises the steps of: obtaining gaze data from a gaze tracking system, the gaze data indicating a location of the user’s gaze relative to the video game’s virtual environment; determining, based on the gaze data, whether the user’s gaze falls outside of a predetermined range of gaze locations at which the first region of the video game’s virtual environment is visible to the user; and if so, carrying out the modifying step.

[0022] Optionally, the method comprises the step of outputting, to the user, a prompt for prompting the user to orient their head such that the first region of the video game’s virtual environment is no longer visible to the user.

[0023] Optionally, the prompt is one or more of: a visual prompt for display to the user; an audio prompt for playback to the user; and a haptic prompt for providing haptic feedback to the user.

[0024] In a second aspect, there is provided a computer program comprising processor-implementable instructions which, when executed by a processor, cause the processor to perform the method of the first aspect.

[0025] In a third aspect, there is provided a non-transitory computer-readable storage medium having stored thereon the computer program of the second aspect.

[0026] In a fourth aspect, there is provided a device comprising a processor and a memory, the memory having stored thereon processor-implementable instructions which, when executed by the processor, cause the processor to perform the method of the first aspect.

[0027] In a fifth aspect, there is provided a system comprising a device according to the fourth aspect, and a first sensor configured to generate orientation data indicating an orientation of the user’s head relative to a datum orientation.

[0028] Optionally, the system comprises a display device for displaying the video game to the user.

[0029] Embodiments of the present description will now be described by way of example with reference to the accompanying drawings, in which:Figure 1 is a flowchart illustrating a computer-implemented method according to embodiments of the present description;Figure 2 schematically illustrates a device according to embodiments of the present description; andFigure 3 schematically illustrates a system according to embodiments of the present description.

[0030] A (computer-implemented) video game processing method, and a device and system for performing the same are disclosed. In the following description, a number of specific details are presented in order to provide a thorough understanding of the embodiments of the present description. It will be apparent, however, to a person skilled in the art that these specific details need not be employed to practice the present invention. Conversely, specific details known to the person skilled in the art are omitted for the purposes of clarity where appropriate.

[0031] Turning now to Figure 1, in embodiments of the present description, a video game processing method comprises the steps of: outputting a video game for display to a user (step S100); obtaining orientation data from a first sensor, the orientation data indicating an orientation of the user’s head relative to a datum orientation (step S102); determining, based on the orientation data, whether the orientation of the user’s head falls outside of a predetermined range of orientations at which a first region of the video game’s virtual environment is visible to the user (step S104); and if so, modifying the outputting of at least the first region of the video game’s virtual environment (step S106).

[0032] This is to say that embodiments of the present description relate to a method of controlling processing of the video game such that poppings and / or pop-ins of virtual elements within a first region of the video game’s virtual environment (that is, modifications to the output of at least that first region) occur when the user’s head is found to be oriented away from the first region (that is, oriented so that the first region is not / less visible to the user). This way, the jarring / discomforting / dangerous visual effect of popping / pop-in is less likely to be noticed by the user, or, if noticed, the extent to which popping / pop-in affects the user is likely to be reduced. This is because the popping / pop-in occurs outside / in the periphery of the user’s field of view.

[0033] Alternatively or in addition, this method of controlling processing of the video game may be carried out such that the video game is paused (that is, the output of at least the first region of the video game’s environment is modified) when the user’s head is found to be oriented away from the first region. This way, the adverse effects of looking away from an ongoing video game may be reduced in terms of frequency and / or extent, as the video game is prevented from continuing when it is found that the user is looking away from the screen.

[0034] Outputting a Video Game

[0035] Given that embodiments of the present description relate to modifying output of at least a first region of a video game’s virtual environment, the video game in question is first output for display.

[0036] Therefore, in embodiments of the present description, the method comprises the step of outputting a video game for display to a user (step S100). Outputting step S100 may be performed by an A / V port (such as an HDMI port, USB port, Ethernet (R) port, Wi-Fi (R) ports, Bluetooth (R) port, and the like) of an entertainment device (such as a video games console, a computing system, and the like), optionally in conjunction with a processing unit (such as a CPU, GPU, and the like) of the entertainment device.

[0037] As will be appreciated, the phrase “outputting the video game for display” should be taken to mean that the image data (image frames, for example) of the video game may be sent / output from the entertainment device (via an A / V port, for example) to a screen, and this screen may subsequently display the image data to the user.

[0038] As will be appreciated, in order to output image data / frames of the video game, such image frames / data may first be rendered. The skilled person knows what rendering image frames entails, and so detailed description regarding typical rendering processes will be omitted for brevity’s sake.

[0039] This rendering may be carried out by the same computing system which carries out the method according to embodiments of the present description. This is to say that embodiments of the present description may be carried out by an entertainment device that renders the video game’s image frames, this rendering being performed by a processing unit (such as a CPU, GPU, and the like) of the entertainment device, for example. Hence more generally, in embodiments of the present description, the method optionally comprises the step of rendering image frames of the video game. In this case, outputting step S100 may comprise outputting the rendered image frames of the video game.

[0040] Alternatively, the rendering may be carried out by a different computing system to that which carries out the method according to embodiments of the present description. For example, embodiments of the present description may be carried out by an entertainment device, but the rendering of the video game’s image frame may be carried out by a processing unit of a server connected to the entertainment device via a network (such as the Internet, for example). In this case, the entertainment device may not be performing the rendering, but rather may receive the rendered image frames of the video game from the server, and output these image frames for display to the user.

[0041] In any case, any type of display screen may be used to display the video game to the user. Examples of such screens shall be outlined below, but skilled persons will appreciate that these examples are entirely non-limiting; skilled persons will understand that other types of display screens are also contemplated within the scope of the present description. Additionally, the skilled person knows what outputting / displaying image frames entails, and so detailed description regarding typical displaying processes will be omitted for brevity’s sake.

[0042] In embodiments of the present description, the outputting step may optionally comprise outputting the at least first region of the video game’s virtual environment to a standalone display screen for display to the user. A “standalone display screen” should be taken to mean that the display screen may be viewed by the user when the device comprising the display screen is mounted to / disposed on a surface (such as a wall, a shelf, a desk, the user’s lap, or the like) and / or held in the user’s hand. At least within the context of the present description, display screens found in wearable devices (such as augmented reality glasses, a head-mounted display system, or the like) would not be considered “standalone display screens”, as such display screens are typically viewed when the user wears the device on their head, rather than when disposed on a surface and / or held by the user. Hence more generally, a standalone display screen may be comprised within a device configured such that, when in use, the device may be mounted to or disposed on a surface, and / or held in the user’s hand.

[0043] Examples of standalone display screens include a television screen, a computer monitor, a laptop screen, a tablet computer screen, a smartphone screen, and a screen of a portable video game console. However, these examples are entirely non-limiting; the skilled person will understand that other types of standalone display screens are also contemplated within the scope of the present description.

[0044] In any case, at least the first region of the video game’s virtual environment (that is, the region which is to be modified when the user is not looking at it) is output to the standalone display screen for display to the user. This is to say that if a standalone display screen is used to display the video game’s virtual environment, then whichever region of the virtual environment is currently being displayed by the standalone display screen is taken to comprise the first (to-be-modified) region. Alternatively put, when the user is found to not be looking at (a part of) the standalone display screen, the region of the virtual environment that is being displayed within (that part of) the standalone display screen may be modified －this region is taken to be the first region; the region of the virtual environment which is being displayed on a (part of a) screen which, due to the user’s current head orientation, is not / less visible to the user.

[0045] For example, the standalone display screen may be displaying a tree of the video game’s environment on the left hand side of the screen. If the user’s head is found to be oriented away from (the left hand side of) the screen so that the left hand side of the screen is not / less visible to the user, then the tree’s level of detail may be modified. While this modification may result in popping, this popping should be less visible (and so less impactful) to the user. As will be appreciated from this non-limiting example, the left hand side of the screen may be thought of as displaying the first region of video game’s virtual environment, this first region comprising the tree that was modified.

[0046] As an alternative to standalone display screens, a head-mounted display (HMD) system may be used to display the video game to the user. As will be appreciated, given the typically fixed positional relationship between the screens of the HMD and the user’s head (by virtue of the user wearing the HMD on their head), the first (to-be-modified) region cannot be defined as being displayed within a (part of the) screen away from which the user’s head is oriented away (as was the case with standalone display screens). This is because, when in use (that is, when worn by the user), the screens of the HMD are positioned in front of the user’s face / eyes regardless of the user’s head orientation, and so there is no (part of a) HMD screen which is not / less visible to the user depending on their head orientation － all of the HMD screens (and so all of the displayed region of the video game’s virtual environment) are visible at all head orientations.

[0047] When an HMD system is used to display a video game, there is typically a determination as to which region of the video game’s virtual environment is to be displayed on the HMD system’s screen, this determination being based on the HMD system’s orientation relative to some datum orientation. It will be appreciated that the HMD system’s orientation is dependent upon the user’s head orientation, given that when the user wears the HMD system, the fixed positional relationship between the HMD system and the user arises. Thus, when in use, a change in the user’s head orientation may ultimately cause a change in the region of the video game’s virtual environment that is displayed on the HMD system’s screens.

[0048] As such, in embodiments where an HMD system is used to display the video game, the first (to-be-modified) region of the video game’s virtual environment may be defined as a region that is not / no longer being displayed on the screens of the HMD system due to (a change in) the user’s head orientation. As will be appreciated, the user’s head orientation may be indicated in orientation data obtained from a sensor such as an inertial measurement unit (IMU) of the HMD system, a standalone camera that captures images of the user wearing the HMD system, or the like.

[0049] Hence more generally, in embodiments of the present description, outputting step S100 may comprise the steps of: obtaining the orientation data from a second sensor (an IMU or standalone camera, for example); determining, based on the orientation data from the second sensor, which region of the video game’s virtual environment is to be output to one or more screens of a head-mounted display, HMD, system; and outputting the determined region of video game’s virtual environment to the one or more screens of the HMD system for display to the user.

[0050] The second sensor may be thought of as the sensor which obtains orientation data for use in a subsequent determination as to which region of the video game’s virtual environment is to be displayed on the HMD system’s screens. Meanwhile, the first sensor may be thought of as the sensor which obtains orientation data for use in a subsequent determination as to whether the first (to-be-modified) region is visible to the user. It will be appreciated that the second sensor may be the same as the first sensor, that is, the orientation data obtained from one sensor may be used for both of the aforementioned determinations. Alternatively, the first sensor and second sensor may be different sensors.

[0051] The step of obtaining orientation data (from either the first sensor or the second sensor) shall be discussed in more detail later herein. The skilled person knows what is meant by a head-orientation-based determination as to which region of video game’s virtual environment is to be displayed on the HMD system’s screens, and so detailed description regarding typical displaying processes for HMD systems will be omitted for brevity’s sake. For example, the skilled person knows that this determination is typically carried out by a processing unit of an entertainment device. Additionally, the skilled person knows what outputting / displaying regions of the video game’s virtual environment entails.

[0052] In any case, when an HMD system is used to display the video game, the first (to-be-modified) region of the video game’s virtual environment may be defined as a region that is not / no longer being displayed on the screens of the HMD system due to (a change in) the user’s head orientation.

[0053] Obtaining Orientation Data

[0054] Given that embodiments of the present description relate to modifying output of at least a first region of a video game’s virtual environment when a user is found to be looking away from the first region, the user’s head orientation is first obtained. Moreover, when an HMD system is used to display the video game, the user’s head orientation is first obtained to determine which region of the video game’s environment is to be displayed to the user.

[0055] Therefore, in embodiments of the present description, the method comprises the step of obtaining orientation data from a first sensor (and / or a second sensor), the orientation data indicating an orientation of the user’s head relative to a datum orientation (step S102). Obtaining step S102 may be performed by an A / V port (such as an HDMI port, USB port, Ethernet (R) port, Wi-Fi (R) ports, Bluetooth (R) port, and the like) of an entertainment device (such as a video games console, a computing system, and the like), optionally in conjunction with a processing unit (such as a CPU, GPU, and the like) of the entertainment device.

[0056] As will be appreciated, the datum orientation may be any orientation of the user’s head relative to the entertainment device outputting the video game. The most commonly used datum orientation may be a “towards the screen” orientation where the user’s face is directed towards the (standalone) display screen displaying the video game to the user. However, other orientations may be set as the datum orientation. In any case, the datum orientation should be taken to mean an orientation relative to which any angular displacements (that is, orientations) of the user’s head are measured by the first sensor and / or second sensor. As such, the user’s head has a zero-degree angular displacement in the case where it is oriented in the datum orientation.

[0057] In any case, in embodiments where a standalone display screen is used to display the video game, the first sensor may optionally comprise an inertial measurement unit, IMU, wearable on the user’s head and configured to measure the orientation of the user’s head.

[0058] As will be appreciated, an IMU is typically used to track the position and / or motion of the device to which it is coupled. In this case, the device in question may be some sort of head-wearable item such as a headband or hat, for example. To facilitate this tracking, the IMU typically comprises at least one of an accelerometer, a gyroscope and a magnetometer. The IMU may be integral to the to-be-tracked device, or may be attached to the device via some (detachable) coupling mechanism. In any case, as the device moves, the accelerometer, gyroscope and / or magnetometer of the IMU may measure kinematic properties such as the position, velocity (rate of change in position), acceleration (rate of change in velocity), and / or jerk (rate of change of acceleration) of the moving device.

[0059] It should be noted that each type of IMU component may be capable of directly measuring a respective (sub)set of kinematic properties. For example, an accelerometer may be capable of directly measuring acceleration, and so to obtain position / velocity, the acceleration measurement data obtained by the accelerometer may be numerically integrated with respect to time. Conversely, to obtain jerk, the acceleration measurement data may be numerically differentiated with respect to time.

[0060] As will be appreciated, the IMU may be coupled to / integrated with the to-be-tracked device, and during use, this IMU may be used to measure an angle / orientation of the peripheral device with respect to some datum orientation thereof by directly measuring kinematic properties (such as rotational velocity / acceleration) and performing suitable data processing steps (such as numerical integration / differentiation). As will be appreciated, the datum orientation of the head-wearable device may be used to define the datum orientation of the user’s head, given that the device is being used to track the movements of the user’s head.

[0061] Optionally, the IMU may be coupled to a head-wearable audio system. A “head-wearable audio system” may be thought of as a system which a user wears on / around their head and which is suitable for outputting audio to a user. The term “coupled to” here should be taken to mean the light sensing element may be integral with the head-wearable audio system, or attachable thereto.

[0062] Examples of head-wearable audio systems include a pair of headphones, a pair of bone conduction headphones, and a pair of earphones. It will be appreciated that these examples are entirely non-limiting; the skilled person will understand that other types of head-wearable audio systems are also contemplated within the scope of the present description.

[0063] In any case, the IMU may transmit orientation data to the input port of the entertainment device via wired or wireless communication methods such as a USB / Ethernet (R) cable, or Wi-Fi (R) / Bluetooth (R) communication. This orientation data may be the angle / orientation of the head-wearable device / system (and so too the angle / orientation of the user’s head). In this case, obtaining step S102 may comprise receiving the orientation data (at an input port of the entertainment device, for example). Alternatively, this orientation data may be some other kinematic property from which the angle / orientation of head-wearable device / system (and so too the user’s head orientation) may be derived. For example, this orientation data may be a rotational acceleration and / or a rotational velocity. In this case, obtaining step S100 may comprise receiving the orientation data (at an input port of the entertainment device, for example), and determining the user’s head orientation based on the orientation data (by using a processing unit of the entertainment device to numerically integrate / differentiate the measurement data, for example).

[0064] Alternatively, in embodiments where a standalone display screen is used to display the video game, the first sensor may optionally comprise a light sensing element wearable on the user’s head and configured to sense light incident thereon and generate orientation data based on one or more illumination characteristics of the sensed light. Examples of light sensing elements include a photodiode, and a camera. However, these examples are entirely non-limiting; the skilled person will understand that other types of light sensing elements are also contemplated within the scope of the present description.

[0065] In any case, the light sensing element may be wearable on the head of the user and configured such that, when in use, the light sensing element senses light from the standalone display screen (outputting the video game) when the face of the user is oriented towards the screen. This is to say that the light sensing element is ideally positioned on the user’s head such that when the user’s head / face is directed towards the standalone display screen, the light sensing element may sense the light emitted from the screen.

[0066] The light sensing element may generate a signal based on the illumination characteristics (brightness / colour) of incident light from given light source. For example, the light sensing element may output electrical signals having increasingly higher voltages in response to detecting increasingly brighter light. In this case, the signal generated by the light sensing element may be thought of as orientation data, as the detected light’s illumination characteristics (on which the signal is based) are dependent on the orientation of the light sensing element relative to the light source. For example, the signals generated by the light sensing element may have increasingly higher voltages as the user’s head (and so too the light sensing element) is gradually oriented towards the standalone display screen due to the gradual increase in the apparent brightness of the light output from the screen. Thus, the signal output by the light sensing element may be used to determine whether the user’s head is oriented towards / away from the standalone display screen, and as such may be considered orientation data. Using the light sensing element’s signal as data indicative of the user’s head orientation (that is, orientation data) shall be discussed later herein.

[0067] In any case, the light sensing element may be a separate head-wearable item to that of the head-wearable audio system － a headband or hat, for example. Alternatively, the light sensing element may be coupled to the head-wearable audio system. The term “coupled to” here should be taken to mean the light sensing element may be integral with the head-wearable audio system, or attachable thereto.

[0068] In any case, the light sensing element may transmit the signal to the input port of the entertainment device via wired or wireless communication methods such as a USB / Ethernet (R) cable, or Wi-Fi (R) / Bluetooth (R) communication. As such, obtaining step S102 may comprise receiving the signal from the light sensing element (at an input port of an entertainment device, for example).

[0069] In embodiments where an HMD system is being used to display the video game, the first sensor may be one of: an IMU of the HMD system, the IMU being configured to measure the orientation of the user’s head; and a standalone camera configured to capture images of the HMD system when worn by the user.

[0070] As will be appreciated, the discussion provided earlier herein regarding measuring the user’s head orientation using IMUs may apply here, mutatis mutandis. As mentioned previously, HMD systems typically comprise an IMU to track the orientation of the user’s head, and subsequently determine which region of the video game’s virtual environment is to be displayed on the HMD system’s screens. As such, it will be appreciated that this in-built IMU of the HMD system may be utilized in obtaining step S102.

[0071] Alternatively, the first sensor may be a standalone camera. A “standalone camera” should be taken to mean that the camera may be mounted to / disposed on a surface (such as a wall, a shelf, a desk, the user’s lap, or the like) and / or held in the user’s hand. At least within the context of the present description, cameras found in wearable devices (such as body cameras, head-mounted cameras, the aforementioned light sensing element, or the like) would not be considered “standalone cameras”, as such cameras are worn on the user’s torso or head, rather than being disposed on a surface and / or held by the user. Hence more generally, a standalone camera may be configured such that, when in use, the standalone camera may be mounted to or disposed on a surface, and / or held in the user’s hand.

[0072] The standalone camera may be used to capture images of the HMD system when worn by the user. As will be appreciated, the captured images may be thought of as orientation data, as such images depict the user’s head (and so too the HMD system) as being oriented in certain directions relative to the camera. To determine the orientation of the user’s head / HMD system from the captured images, various computer vision algorithms and / or optical tracking methods may be employed. The details of such algorithms / methods are known to the skilled person, and so are omitted for brevity’s sake. For example, the skilled person knows that such algorithms / methods may be carried out using a processing unit of the entertainment device.

[0073] Further, it will be appreciated that a standalone camera may be used as the first sensor in the case where a standalone display screen is being used to display the video game － instead of imaging the HMD system when worn by the user, the camera captures images of the user’s head, and computer vision / optical tracking may be employed to determine the user’s head orientation (as captured in the images).

[0074] In any case, it should be noted that a “towards the screen” orientation may not be suitable as a datum orientation when an HMD system is being used. This is because the screens of the HMD system are at fixed position relative to the user’s eyes / head, and so all orientations of the user’s head would be considered a “towards the screen” orientation. In this case, a more suitable datum orientation may be a “towards the device / camera” orientation, in which the user’s head is oriented towards the entertainment device outputting the video game or, if used, towards the standalone camera capturing images of the user wearing the HMD system.

[0075] In any case, the IMU of the HMD system or the standalone camera may transmit the orientation data to the input port of the entertainment device via wired or wireless communication methods such as a USB / Ethernet (R) cable, or Wi-Fi (R) / Bluetooth (R) communication. As such, obtaining step S102 may comprise receiving the orientation data from the IMU of the HMD system (at an input port of an entertainment device, for example). Alternatively, obtaining step S102 may comprise receiving, as the orientation data, images captured by the standalone camera (at an input port of an entertainment device, for example), and determining the user’s head orientation based on the received images (using a computer vision algorithm, optical tracking method, or the like).

[0076] In embodiments where an HMD system is used to display the video game, the orientation data (indicative of the user’s head orientation) may be obtained from a second sensor. As mentioned previously, the second sensor may be thought of as the sensor which obtains orientation data for determining the region to be displayed on the HMD system’s screens, whereas the first sensor may be thought of as the sensor which obtains orientation data for determining whether the first (to-be-modified) region is visible to the user.

[0077] As will be appreciated, the second sensor may be different from the first sensor. For example, the second sensor may be the other of the IMU (of the HMD system) and the standalone camera. For example, if the first sensor is the IMU, then the second sensor may be the standalone camera. Alternatively, the second sensor may be the same as the first sensor. For example, both the first and second sensors may be embodied by the IMU of the HMD system.

[0078] In any case, once the orientation data has been obtained, the determination as to whether the user’s head is oriented away from the first (to-be-modified) region of the video game’s virtual environment.

[0079] Determining Visibility of First Region

[0080] In embodiments of the present description, the method comprises the step of determining, based on the orientation data, whether the orientation of the user’s head falls outside of a predetermined range of orientations at which a first region of the video game’s virtual environment is visible to the user (step S104). Determining step S104 may be performed by a processing unit (such as a CPU, GPU, and the like) of an entertainment device (such as a video games console, a computing system, and the like).

[0081] The predetermined range of orientations may be thought of as those orientations of the user’s head which enable the user to view the first (to-be-modified) region of the video game’s virtual environment. Given that embodiments of the present description aim to modify the first region when it is not / no longer visible to the user, it naturally follows that such modification(s) should be carried out when the user’s head orientation (as indicated in the obtained orientation data) is determined to fall outside the predetermined range of orientations.

[0082] In embodiments where a standalone display screen is used to display the video game, the predetermined range of orientations may be determined prior to gameplay (that is, predetermined) by carrying out a calibration test while the user’s head orientation (that is, orientation data) is obtained using the first sensor. For example, the user may be instructed to rotate their head in a certain direction until (a part of) the standalone display screen is no longer visible, and provide a confirmation signal (via a video game controller, for example) to the entertainment device to confirm that the (part of the) screen is not visible. Subsequently, the entertainment device stores the orientation data obtained at / proximate to the moment in time at which the confirmation signal is obtained. This process may be repeated, each new process instructing the user to rotate their head in a different direction. The plurality of stored pieces of orientation data may be used to determine a range of orientations at which the user is able to view the screen. For example, an interpolation between the (discrete) pieces of orientation data may be carried out to determine a locus of orientations, the range contained within this locus being the predetermined range of orientations.

[0083] In embodiments where the first sensor comprises a light sensing element, determining step S104 may comprise determining whether the orientation data fails to satisfy any of one or more illumination criteria, modifying step S106 (discussed later herein) being carried out if so.

[0084] As mentioned previously, this light sensing element may output a signal (that is, orientation data) based on the illumination characteristics of a given light source’s outputted light (outputting electrical signals having increasingly higher voltages in response to detecting increasingly brighter light, for example). As such, the signal voltage (and / or other signal parameter for that matter) may be used as a proxy for the detected light source’s illumination characteristics. Therefore, “illumination criteria” should be taken to mean criteria relating to parameters of the signal sent by the light sensing element, such parameters being proxies for the illumination characteristics of the light source whose light was detected by the light sensing element.

[0085] As will be appreciated, the illumination criteria may be defined such that they are satisfied by a signal (that is, orientation data) generated and sent in response to detecting light output from (a part of) the standalone display screen outputting the video game, and are not satisfied otherwise, for example. As will be appreciated, the aforementioned calibration test may be carried out (using the light sensing element) to define these illumination criteria, for example. Once the illumination criteria are define, determining whether the (part of the) display screen is not / no longer visible to the user (that is, determining step S104) may be carried out by determining whether the signal received from the light sensing element fails to satisfy any of the illumination criteria.

[0086] Examples of illumination criteria include a (change in) light colour, a (change in) light intensity (also known as brightness), and a frequency range or wavelength range of the light (such as a wavelength range corresponding to infra-red light or visible light, for example). It will be appreciated that these examples are entirely non-limiting; the skilled person will understand that other types of illumination criteria are also contemplated within the scope of the present description. It will also be appreciated that while the examples given explicitly refer to the illumination characteristics of the detected light, these same examples also implicitly relate to parameters of the signal received from the light sensing element given the aforementioned correspondence between the light detected and signal generated by the light sensing element.

[0087] Given the dynamic nature of video games, the light output from the standalone display screen displaying the video game may at times fail to satisfy the illumination criteria. For example, if a certain level of the video game involves exploring a dark environment (a cave, a haunted house, or the like), then the light output from the standalone display screen may fail to satisfy an illumination criteria relating to brightness, thereby causing a false positive determination that the user is looking away from the screen when they are actually looking at the screen.

[0088] To reduce the frequency / occurrence of these false positive determinations, calibration frames may be interspersed amongst image frames of the video game. A calibration frame should be taken to mean an image frame whose image data is such that, when output on a standalone display screen, the resulting light output from (a part of) the screen causes the light sensing element (directed towards the screen) to output a signal which satisfies one or more of the illumination criteria. In this sense, it will be appreciated that a calibration frame helps to calibrate the light sensing element － the light sensing element generates and sends a signal based on the light output from the screen (due to displaying the calibration frame), and the entertainment device finds that this signal satisfies one or more illumination criteria, and so no (false positive) determination is carried out. This to say that, due to the displaying of the calibration frames, the light sensing element is more likely to provide signals which fail to satisfy the illumination criteria when the user’s head (and so too the light sensing element) is oriented away from (a part of) the standalone display screen.

[0089] The image data of such calibration frames may be distinct from the video game’s image frames, resulting in a sharp visual contrast (change in colours / brightnesses / frequencies / wavelengths of light) whenever a calibration frame is output either before or after a given video game image frame. Given that frame rates of displaying video game typically exceed 50 frames per second, such contrasts between video game image frames and calibration frames are typically not detected by users, and so the use of such calibration frames should not interfere with the user’s experience of the video game. However, light sensing elements are able to detect such calibration frames, despite them only being displayed for less than or equal to 0.02 seconds each.

[0090] Hence more generally, in embodiments of the present description, the method may optionally comprise the steps of: obtaining rendered image frames of the video game; inserting a calibration frame after every N image frames of the video game have been rendered, wherein N ≧ 1, wherein each inserted calibration frame comprises predetermined image data for satisfying one or more of the illumination criteria; and outputting the video game having the inserted calibration frames.

[0091] The obtaining rendered image frames may be performed by an input port (such as an HDMI port, USB port, Ethernet (R) port, Wi-Fi (R) ports, Bluetooth (R) port, and the like) of an entertainment device (such as a video games console, a computing system, and the like), for example. Optionally, this obtaining may comprise rendering the image frames, in which case a processing unit (a CPU, GPU, and the like) of the entertainment device (such as a video games console, a computing system, and the like) may be employed to render the image frames of the video game. The inserting step may be performed by a processing unit of an entertainment device, for example. The outputting step may be performed by an A / V port (such as an HDMI port, USB port, Ethernet (R) port, Wi-Fi (R) ports, Bluetooth (R) port, and the like) of the entertainment device, optionally in conjunction with a processing unit of the entertainment device.

[0092] The skilled person knows what rendering image frames entails, and so detailed description regarding typical rendering processes will be omitted for brevity’s sake. Regarding the inserting step, the calibration frame may be stored within a buffer within which rendered (that is, to-be-output) image frames (of the video game) are stored, for example. Such a buffer typically stores the image frames in the order in which they are to be output to a display screen, and so in order to insert the calibration frame after every N image frames, each calibration frame is stored within the buffer at a memory location corresponding to the (N+1)th, 2(N+1)th, 3(N+1)th, and so on, frame to be output to the display screen, for example. The skilled person knows what outputting image frames entails, and so detailed description regarding typical displaying processes will be omitted for brevity’s sake.

[0093] In embodiments where an HMD system is being used to display the video game, the predetermined range of orientations may be defined as those orientations of the user’s head (and so too the HMD system) which cause the first region of the video game’s virtual environment to be displayed on the screen(s) of the HMD system. Thus, the user’s head orientation may be deemed as falling outside the predetermined range of orientations when the screens of the HMD system do not display the first region.

[0094] In any case, once it is determined that the user’s head is oriented such that the first region is not / no longer visible to the user, the outputting of the first region may be modified.

[0095] Modifying Output of First Region

[0096] As mentioned previously, poppings and pop-ins of virtual elements may be jarring / discomforting / dangerous for a user to view, especially if they occur frequently and / or affect virtual elements which take up increasingly larger amounts of screen space. To alleviate or mitigate this issue, the processing of the video game may be controlled such that poppings and / or pop-ins of virtual elements within a first region of the video game’s virtual environment (that is, modifications to the output of at least that first region) occur when the user’s head is found to be oriented away from the first region.

[0097] Additionally, the processing of the video game may be controlled such that the video game is paused, when the user’s head is found to be oriented away from the first region, thereby reducing the frequency and / or extent of the adverse effects of looking away from an ongoing video game (such as the user’s virtual character being injured / killed, for example).

[0098] Therefore, in embodiments of the present description, the method comprises the step of modifying the outputting of at least the first region of the video game’s virtual environment (step S106), this modifying step S106 being carried out if the orientation of the user’s head falls outside of a predetermined range of orientations. Modifying step S106 may be performed by a processing unit (a CPU, GPU, and the like) of the entertainment device (such as a video games console, a computing system, and the like).

[0099] Modifying step S106 may optionally comprise initiating a game event within the at least first region of the video game’s virtual environment. Examples of initiating game events include pausing the video game; instantiating a virtual element within the first region of the video game’s virtual environment (that is, carrying out a pop-in process); modifying a state or attribute of an instantiated virtual element that is located within the first region of the video game’s virtual environment (that is, carrying out a popping process); and outputting an audio signal and / or haptic signal corresponding to an instantiated virtual element that is located within the first region of the video game’s virtual environment (a “leaf rustling” sound associated with a virtual tree contained within the first region, for example). These examples are entirely non-limiting; skilled persons will understand that other types of game events are also contemplated within the scope of the present description.

[0100] Regarding the popping process, a modifiable state or attribute of a virtual element may be a colour / texture / level of detail / size / shape of the virtual element, a shading / rendering process used on the virtual element, a health / stamina / magic / power level of the virtual element, or the like.

[0101] In any case, modifying step S106 is carried out on the first region when it is found that (and while the) first region is not / no longer visible to the user.

[0102] As mentioned previously, by controlling the popping / pop-in such that it occurs outside / in the periphery of the user’s field of view, the jarring / discomforting / dangerous visual effect of popping / pop-in is less likely to be noticed by the user, or, if noticed, the extent to which popping / pop-in affects the user is likely to be reduced.

[0103] In certain cases, this process may result in a startling effect for the user, which may be desirable to enhance the gameplay experience. For example, while playing a horror game, a user may look away from a standalone display screen outputting the video game (to talk to a friend, look at a smartphone, or the like). When it is found that the user is looking away from the screen, a pop-in process may be performed to instantiate a virtual monster / ghost / zombie within the displaying region (that is, first region) of the video game’s virtual environment. In this case, a dual benefit arises in that the pop-in process has less of a visually jarring effect on the user (due to, say, a sharp change in screen brightness when the dark room is brightened by the sudden appearance of the virtual ghost), and the pop-in produces a starling “jump-scare” effect befitting of horror games. This is to say that the user is more likely to react in fear to the presence of the ghost, rather than feeling a jarring discomfort. The same can be said of initiating an audio / haptic signal (or, more generally, a game event) when the user is looking away from the screen.

[0104] Various Embodiments

[0105] As will be appreciated, a user may still see the first region of the video game’s virtual environment even though their head orientation is found to suggest otherwise. This is because the user may still orient their gaze towards the first region while orienting their head away from it (the user may move their head to direct their speech towards a friend but still view the first region). In this case, there may be a false positive determination that the user is looking away from the screen when they are actually looking at the screen.

[0106] To reduce the frequency / occurrence of false positive determinations, it may be beneficial to use an additional gaze tracking process to ensure that the user’s gaze (as well as head) is oriented / directed away from the first region of the video game’s virtual environment.

[0107] Hence more generally, in embodiments of the present description, the method may optionally comprise the steps of: obtaining gaze data from a gaze tracking system, the gaze data indicating a location of the user’s gaze relative to the video game’s virtual environment; determining, based on the gaze data, whether the user’s gaze falls outside of a predetermined range of gaze locations at which the first region of the video game’s virtual environment is visible to the user; and if so, carrying out modifying step S106.

[0108] The skilled person knows what gaze tracking entails, and so detailed description regarding typical gaze tracking processes will be omitted for brevity’s sake. For example, the skilled person knows that a gaze tracking system may comprise a camera configured to capture images of the user’s eye(s), and computer vision / optical tracking may be used to determine the user’s gaze direction (that is, gaze data). The skilled person also knows that gaze tracking system may be standalone, or may be comprised within HMD systems.

[0109] The predetermined range of gaze locations may be determined before gameplay (that is, predetermined) using a calibration test similar to that discussed above with respect to the predetermined range of (head) orientations, for example. Modifying step S106 has been discussed earlier herein.

[0110] In embodiments employing additional gaze tracking, modifications to the first region of the video game’s virtual environment are carried out when both the user’s head and gaze are oriented / directed away from the first region of the video game’s virtual environment. As such, the frequency / occurrence of false positives may be reduced.

[0111] Certain users may look away from the first region infrequently. Given that embodiments of the present description aim to prevent poppings / pop-ins of virtual elements occurring until the user looks away, the visual appearance of the first region may be updated at a slower rate compared to users who look away from the first region more regularly. This slower update rate may result in decreased user satisfaction － the virtual elements within the first region may be displayed at a lower resolution / level of detail than preferable for longer periods of time, for example.

[0112] To alleviate this issue, the user may be prompted to look away from the first region, thereby providing an opportunity to carry out the poppings / pop-ins, and so update the visual appearance of the video game more frequently.

[0113] Hence more generally, in embodiments of the present description, the method may optionally comprise the step of outputting, to the user, a prompt for prompting the user to orient their head such that the first region of the video game’s virtual environment is no longer visible to the user. This outputting step may be performed by an A / V port of the entertainment device, optionally in conjunction with a processing unit of the entertainment device.

[0114] As will be appreciated, the prompt may be one or more of: visual prompt for display to the user (a vignette effect along an edge of the standalone display screen, for example), an audio prompt for playback to the user (an in-game sound effect such as a bird call, for example), a haptic prompt for providing haptic feedback to the user (via haptic actuators of a game controller). These examples are entirely non-limiting; skilled persons will understand that other types of prompts are also contemplated within the scope of the present description.

[0115] The skilled person knows what is meant by outputting prompts to users, and so detail discussion on typical prompting / notification processes will be omitted for brevity’s sake.

[0116] Computer Program and Storage Medium

[0117] It will be appreciated that the above methods may be carried out on conventional hardware (such as computing system 1 of Figure 2) suitably adapted as applicable by software instruction or by the inclusion or substitution of dedicated hardware.

[0118] Thus the required adaptation to existing parts of a conventional equivalent device may be implemented in the form of a computer program product comprising processor implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, solid state disk, PROM, RAM, flash memory or any combination of these or other storage media, or realised in hardware as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array) or other configurable circuit suitable to use in adapting the conventional equivalent device. Separately, such a computer program may be transmitted via data signals on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0119] Device and System

[0120] In embodiments of the present description, a device comprises a processor and a memory, the memory having stored thereon processor-implementable instructions which, when executed by the processor, cause the processor to perform embodiments of the above-described method.

[0121] Turning now to Figure 2, as an example embodiment of the present description, the device may be computing system 1. The circuitry of computing system 1 may comprise the following components listed in turn below.

[0122] Computing system 1 may comprise processing unit 10. Processing unit 10 may be a central processing unit (CPU) and / or a graphical processing unit (GPU). The CPU may be a single or multi core processor. The GPU may be physically separate to the CPU, or may be integrated with the CPU as a system on a chip (SoC). Processing unit 10 may be configured to execute processor-implementable instructions which cause it to carry out embodiments of the above-described method.

[0123] Computing system 1 may comprise memory 20. Memory 20 may be a RAM, ROM, and / or the like. The RAM may be physically separate to the CPU and / GPU, or may be integrated therewith as part of an SoC. Alternatively or in addition, memory 20 may be an external or internal hard drive, or an external or internal solid state drive. Memory 20 may be configured to store the processor-implementable instructions.

[0124] Computing system 1 may comprise A / V output port 30. A / V output port 30 may enable computing system 1 to transmit audio / visual outputs to one or more other devices / systems. Examples of A / V output port 30 include HDMI ports, USB ports, Ethernet (R) ports, Wi-Fi (R) ports, Bluetooth (R) ports, and the like.

[0125] Computing system 1 may comprise input port 40. Input port 40 may enable computing system 1 to receive data from one or more other devices / systems. Examples of Input port 40 include USB ports, Ethernet (R) ports, Wi-Fi (R) ports, Bluetooth (R) ports, and the like.

[0126] Where components of computing system 1 are not integrated, such components may be connected either by a dedicated data link or via an I / O bus.

[0127] It will be apparent to a person skilled in the art that variations in the operations of the above device corresponding to the various embodiments of the computer-implemented method as described and claimed herein are considered within the scope of the present invention.

[0128] Turning now to Figure 3, in embodiments of the present description, a system comprises the above-described device (such as computing system 1), and first sensor 2 configured to generate orientation data indicating an orientation of the user’s head relative to a datum orientation.

[0129] Optionally, the system may comprise display device 3 for displaying the video game to the user. Examples of display device 3 include a standalone display screen, and a HMD system, both of which have been discussed elsewhere herein.

[0130] The foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting of the scope of the invention, as well as other claims. The disclosure, including any readily discernible variants of the teachings herein, defines, in part, the scope of the foregoing claim terminology such that no inventive subject matter is dedicated to the public.

Claims

1. A video game processing method, comprising the steps of: outputting a video game for display to a user; obtaining orientation data from a first sensor, the orientation data indicating an orientation of the user’s head relative to a datum orientation; determining, based on the orientation data, whether the orientation of the user’s head falls outside of a predetermined range of orientations at which a first region of the video game’s virtual environment is visible to the user; and if so, modifying the outputting of at least the first region of the video game’s virtual environment.

2. A method according to claim 1, wherein the modifying step comprises initiating a game event within the at least first region of the video game’s virtual environment.

3. A method according to claim 2, wherein initiating the game event comprises one or more of: i. pausing the video game; ii. instantiating a virtual element within the first region of the video game’s virtual environment; iii. modifying a state or attribute of an instantiated virtual element that is located within the first region of the video game’s virtual environment; and iv. outputting an audio signal and / or haptic signal corresponding to an instantiated virtual element that is located within the first region of the video game’s virtual environment.

4. A method according to any preceding claim, wherein the outputting step comprises outputting the at least first region of the video game’s virtual environment to a standalone display screen for display to the user.

5. A method according to claim 4, wherein the standalone display screen is one of: i. a television screen; ii. a computer monitor; iii. a laptop screen; iv. a tablet computer screen; v. a smartphone screen; and vi. a screen of a portable video game console.

6. A method according to claim 4 or claim 5, wherein the first sensor comprises an inertial measurement unit, IMU, wearable on the user’s head and configured to measure the orientation of the user’s head.

7. A method according to claim 6, wherein the IMU is coupled to a head-wearable audio system.

8. A method according to claim 7, wherein the head-wearable audio system is one of: i. a pair of headphones; ii. a pair of bone conduction headphones; and iii. a pair of earphones.

9. A method according to claim 4 or claim 5, wherein: the first sensor comprises a light sensing element wearable on the user’s head and configured to sense light incident thereon and generate orientation data based on one or more illumination characteristics of the sensed light; and the determining step comprises determining whether the orientation data fails to satisfy any of one or more illumination criteria, the modifying step being carried out if so.

10. A method according to claim 9, wherein the light sensing element is coupled to a head-wearable audio system.

11. A method according to claim 9 or claim 10, wherein the light sensing element comprises one or more of: i. a photodiode; and ii. a camera.

12. A method according to any one of claims 9 to 11, wherein the one or more illumination criteria comprise one or more of: i. a light colour; ii. a light intensity; iii. a change in light colour; iv. a change in light intensity; and v. a frequency range or wavelength range of the light.

13. A method according to any one of claims 9 to 12, comprising the steps of: obtaining rendered image frames of the video game; inserting a calibration frame after every N rendered image frames of the video game, wherein N ≧ 1, wherein each inserted calibration frame comprises predetermined image data for satisfying one or more of the illumination criteria; and outputting the video game having the inserted calibration frames.

14. A method according to any one of claims 1 to 3, wherein the outputting step comprises: obtaining the orientation data from a second sensor; determining, based on the orientation data from the second sensor, which region of the video game’s virtual environment is to be output to one or more screens of a head-mounted display, HMD, system; and outputting the determined region of video game’s virtual environment to the one or more screens of the HMD system for display to the user.

15. A method according to claim 14, wherein the first sensor is one of: i. an inertial measurement unit, IMU, of the HMD system, the IMU being configured to measure the orientation of the user’s head; and ii. a standalone camera configured to capture images of the HMD system when worn by the user.

16. A method according to claim 15, wherein the second sensor is the other of: i. the IMU; and ii. the standalone camera.

17. A method according to claim 14 or claim 15, wherein the second sensor is the same as the first sensor.

18. A method according to any preceding claim, comprising the steps of: obtaining gaze data from a gaze tracking system, the gaze data indicating a location of the user’s gaze relative to the video game’s virtual environment; determining, based on the gaze data, whether the user’s gaze falls outside of a predetermined range of gaze locations at which the first region of the video game’s virtual environment is visible to the user; and if so, carrying out the modifying step.

19. A method according to any preceding claim, comprising the step of outputting, to the user, a prompt for prompting the user to orient their head such that the first region of the video game’s virtual environment is no longer visible to the user.

20. A method according to claim 19, wherein the prompt is one or more of: i. a visual prompt for display to the user; ii. an audio prompt for playback to the user; and iii. a haptic prompt for providing haptic feedback to the user.

21. A computer program comprising processor-implementable instructions which, when executed by a processor, cause the processor to perform the method of any preceding claim.

22. A non-transitory computer-readable storage medium having stored thereon the computer program of claim 21.

23. A device comprising a processor and a memory, the memory having stored thereon processor-implementable instructions which, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 20.

24. A system, comprising: a device according to claim 23; and a first sensor configured to generate orientation data indicating an orientation of the user’s head relative to a datum orientation.

25. A system according to claim 24, comprising a display device for displaying the video game to the user.

Images