Computer game controller tracking system

Abstract

At least one computer game controller device, the or each computer game controller device including an electromagnetic radiation detecting device serving to detect electromagnetic radiation in the invisible spectrum and one or more data processing devices, an image projection screen including an array of marker devices arranged to be illuminated by electromagnetic radiation in the invisible spectrum, and a calibration device including a further data processing device and one or more electromagnetic detecting devices serving to detect electromagnetic radiation in both the visible and invisible spectrum, the calibration device serving to calculate positional offsets of the array of marker devices relative to an image projected on the projection screen.

Description

[0001] Computer Game Controller Tracking System

[0002] The present invention relates to the tracking of a computer game controller, particularly a computer game controller in the form of a gun for use in game play.

[0003] According to one aspect of the present invention, there is provided apparatus comprising:

[0004] • at least one computer game controller device, the or each computer game controller device including an electromagnetic radiation sensing device and one or more data processing devices,

[0005] • an image projection surface including an array of marker devices detectable by the electromagnetic radiation sensing device of the at least one computer game controller device, and

[0006] • a calibration device including a further data processing device and one or more electromagnetic sensing devices serving to detect electromagnetic radiation, wherein the calibration device serves to calculate positional offsets of the array of marker devices relative to an image displayed on the projection surface.

[0007] According to a second aspect of the present invention, there is provided a method of tracking at least one computer game controller device relative to an array of marker devices in a detection space, the method comprising:

[0008] • displaying a visible image on an image projection surface, the image projection surface including the array of marker devices,

[0009] • detecting electromagnetic radiation reflected from the markers enabling detection of the array of markers and processing data relating thereto,

[0010] • utilising the data from the electromagnetic radiation and detection of the visible image to calculate location data of each marker in the array of markers relative to the detection space,

[0011] • calculating any positional offsets of the array of marker devices relative to the image displayed on the projection surface, and

[0012] communicating the location data to the at least one computer game controller device to calculate its own position relative the location of each marker in the detection space. Owing to these aspects, the at least one computer game controller device calculates its position relative to the projection surface with the array of marker devices allowing for consistent positional calculations.

[0013] In order that the invention can be clearly and completely disclosed, reference is now made, by way of example only, to the accompanying drawing, which shows a schematic architecture for a computer game system where tracking of the game controller is necessary for a realistic gaming experience.

[0014] Referring to Figure 1 , a tracking system 2 for the tracking of at least one computer game controller device comprises at least one computer game controller device 4, preferably in the form of a gun-type pointer controller (a so-called smart-gun), the or each game controller device 4 having an integrated sensing device 6 serving to detect electromagnetic radiation, preferably in the invisible spectrum. In that respect, the sensing device 6 is advantageously an infrared (IR) camera device and the game controller device 4 further includes an on-board data processing device (not shown) in the form of one or more microcontrollers. The on-board data processing device manages input data from various hardware elements located in a three-dimensional detection space or game environment and provides signals to produce responsive system feedback to ensure precise control and efficient real-time data processing.

[0015] An image projection surface 8, advantageously in the form of a projection screen, is arranged to have a game image projected thereon and the screen 8 also includes an array of marker devices 10 which array is preferably concealed from view of a game player. The image projection surface may also be a surface from which an image is emitted, such as a television screen (of various varieties) rather than projected on to. The projection surface 8 may also be more three-dimensional compared to a projection screen to improve the perceived depth of the gaming environment or scene.

[0016] The array of marker devices 10 is preferably in the form of a grid pattern and the marker devices are arranged to reflect light, for example, IR light or alternatively the markers 10 could be projected on to the projection surface by an IR image projector device. Physical marker devices 10 preferably comprise IR reflective material, illuminated possibly by an IR blaster or bright IR light source.

[0017] The array of marker devices 10 allows the or each controller device 4 to calculate its position relative to the projection surface 8. The layout pattern of the array 10 allows for consistent positional calculations regardless of its specific arrangement, whether directly on or located behind the surface 8. If the array is located behind the surface 8, the surface is one which is transparent with regard to electromagnetic radiation in the IR spectrum. IR illumination of the array of markers 10 enables the array to be detectable by the sensing device 6 of the or each game controller 4, facilitating accurate localization during gameplay.

[0018] The or each game controller 4 is uniquely identifiable and continuously transmits positional data (relative to the surface 8) and gameplay data (for example, firing data if the controller is a smart-gun).A dedicated communication protocol between the or each game controller 4 and a central data processing device or game server 12, such as a computer having appropriate software, ensures robust data exchange, capturing and relaying operational data for the or each game controller 4 for precise monitoring and control.

[0019] The or each game controller 4 in the form of a smart-gun includes the following components in addition to the sensing device 6 and on-board microcontrollers:

[0020] ■ a plurality of micro switches (for trigger and reload input status)

[0021] ■ at least one solenoid device (for recoil effect)

[0022] ■ at least one solenoid device Mosfet Power Control circuit (for solenoid device control)

[0023] ■ a data cable (for example a Ethernet Hybrid Cat5) for connecting the smartgun to the game server 12

[0024] ■ an addressable multicolour LED to denote system status to give visual feedback for debugging the state of the smart gun (for example, changing colour to signify if connection to the server has been lost)

[0025] ■ an optical cast IR long pass filter The or each game controller 4 may also, depending upon the particular use, include a USB audio adapter with speaker-headphone and microphone connection port and a passive load speaker device.

[0026] The projector surface 8 may include a frame but includes no additional sensors or other hardware apart from the array of markers 10.

[0027] The system also includes a calibration device 14, advantageously in the form of an automatic-calibration device, which includes its own separate data processing device in the form of one or more further microcontrollers and one or more electromagnetic sensing devices (for example, one or more camera devices) serving to detect electromagnetic radiation in both the visible and invisible spectrum. Where, for instance, a projection device, and any enclosure for the projection device (if required) and the automatic-calibration unit is positioned relatively close to the projection surface, first and second cameras are positioned a distance apart, e.g. substantially 1m apart would be needed at that close proximity to the projection surface to capture the entire screen in two parts. Subsequently, by splicing the two images together into one results in the automatic calibration. Where the projection device and any necessary enclosure is located further away from the projection surface; one camera would suffice. With a relatively very large projection surface, and / or a curved projection surface, and / or multiple projection surfaces arranged end-to-end, a plurality of cameras and / or automatic-calibration devices would be required.

[0028] The calibration device 14 calculates any offset data of the array of markers 10 relative to the projected image on the projection surface 8, enabling compensating for variance in 3D / real world positioning changes or image projector device skew and / or manufacturing / construction variances. The calibration device 14 can be automatically run at system startup or manually anytime via an operator dashboard, thereby detecting and compensating for errors, such as faulty or missing markers 10, and ensuring scalability with any arrangement and number of the markers 10. This ensures that the aiming of the smart gun 4 is consistently accurate with respect to the game image on the surface 8. The or each game controller 4 interacts within the system 2 through continuous positional calculations and receives feedback data or commands during gameplay. They can be arranged to receive software updates via the central data processing device 12 and calibration data updates during runtime. During gameplay, the central data processing device 12 sends data to the game controller 4 in response to data sent from the game controller itself, enabling sound effects or LED signals to be emitted from the game controller 4. For instance, if the game controller 4 were to include a loud speaker device, then after pulling a trigger of a smart-gun 4, the system 2 could immediately respond with a 'shot fired' and / or 'out of ammo' sound effect based on ammo status for the player to hear. Other audio and / or visual cues may signal various events during gameplay and aid in identifying and resolving errors (or ‘bugs’) in the software running within the system 2, such as connection status indicated by chimes and / or LED colour changes.

[0029] Data transmitted between the or each game controller 4 and the central data processing device 12 includes positional data, shot data, and reloading status. The calibration device 14 can be controlled via the operator dashboard, with calibration data sent to the central data processing device 12 and forwarded to the or each game controller 4 for immediate positional accuracy updates.

[0030] The calibration device 14 detects both IR (invisible electromagnetic radiation) reflected from the markers and the projected image on the projection screen (visible electromagnetic radiation) to calculate precise location data of each marker in the array 10 relative to the detection space. This precise location data is then communicated to the game controller via the central data processing device 12 so it can correctly calculate its own position relative the exact locations of each marker in the detection space.

[0031] The system 2 uses a combination of computer vision techniques and marker detection algorithms to process the data collected by the or each game controller device 4. The detection of the markers 10, such as fiducial ArUco markers, allows for the determination of the position and orientation of the or each game controller device 4 relative to the projection surface 8. This process includes the following steps: ■ Image Acquisition: The IR-sensing device 6 on each game controller device 4 captures images of the detection space, and in particular detecting the IR- illuminated array of markers 10.

[0032] ■ Pre-processing: The IR long pass filter of each game controller device 4 is used to reduce the amount of extraneous information visible to the IR-sensing device 6, which enhances the efficiency of a detection algorithm.

[0033] ■ Marker Detection: The images are analysed by the detection algorithm to identify the markers 10.

[0034] ■ Position Calculation: This includes detecting the specific pattern of the markers 10, which can then be used to calculate the position and orientation of the or each game controller device 4 relative to the markers. Custom algorithms process the detected marker data to determine the precise position and orientation of the or each game controller device 4. These custom algorithms account for factors such as movement speed, direction, and obstacles in the detection space as well as compensating for the inaccuracies and noise naturally inherent in computer vision technologies

[0035] ■ Real-time Feedback: The processed data is used to provide real-time feedback to the system 2, ensuring accurate and responsive tracking.

[0036] To ensure real-time processing and minimize latency, the system 2 employs several techniques, including:

[0037] ■ Optimized settings for the sensing device 6 to enhance detection speed. ■ Efficient pre-processing of images before passing them to the detection algorithm.

[0038] ■ The custom algorithms being designed for fast and accurate position calculation.

[0039] ■ Decentralization of processing tasks through the use of the microcontroller devices on-board the or each game controller device 4, allowing the system 2 to scale its operation without performance penalties.

[0040] A player is able to interact with the game through a separate touchscreen interface, allowing them to select and modify a game. Customization options include entering their name, taking a picture, and choosing teams for relevant game modes. Potential additional features include extra smart-guns for smaller players and real-time crosshair positional display, configurable per player.

[0041] The system 2 therefore provides versatile tracking capabilities, such as:

[0042] ■ T racking on relatively large projector screens with the markers 10 located behind the surface 8.

[0043] ■ Tracking on relatively smaller screens, like TVs, with the markers 10 located around an outer peripheral zone of the screen.

[0044] ■ Tracking across multiple screens, regardless of their 3D space positioning. ■ Tracking on non-flat surfaces, such as curved projection screens.

[0045] Such versatility is achieved through a configuration file provided to the calibration device 14. The configuration file contains information such as the number or markers in the array, their relative positions in a 3D detection space, their specific IDs, the spatial relationships each marker has within the 3D detection space, and relationships between separate projection surfaces, where present, and each surface’s markers (if multiple screens are being tracked). The configuration file also sets optimized parameters such as real world marker size, lighting conditions, perspective distortion due to players being stood off-centre. There are also settings pertaining to image pre-processing to allow optimisation of marker detection in different locales, e.g. indoors where lighting conditions may be relatively dark, and outdoors where lighting conditions may be relatively light.

Claims

CLAIMS1. Apparatus comprising:• at least one computer game controller device, the or each computer game controller device including an electromagnetic radiation sensing device serving to detect electromagnetic radiation in the invisible spectrum and one or more data processing devices,• an image projection surface including an array of marker devices detectable by the electromagnetic radiation sensing device of the at least one computer game controller device, and• a calibration device including a further data processing device and one or more further electromagnetic sensing device serving to detect electromagnetic radiation in the invisible spectrum, wherein the calibration device serves to calculate positional offsets of the array of marker devices relative to an image displayed on the projection surface.

2. Apparatus according to claim 1 , wherein the computer game controller device is a gun-type pointer controller.

3. Apparatus according to claim 1 or 2, wherein the electromagnetic radiation sensing device is an infrared camera device.

4. Apparatus according to any preceding claim, and further comprising an on-board data processing device in the form of one or more microcontrollers, the on-board data processing device serving to manage input data from various hardware elements located in a three-dimensional detection space and to provide signals to produce responsive system feedback.

5. Apparatus according to any preceding claim, wherein the projection surface is a projection screen arranged to have a game image projected thereon.

6. Apparatus according to any preceding claim, wherein the array is a grid pattern of marker devices, the marker devices arranged to reflect infrared light.

7. Apparatus according to any preceding claim, wherein the marker devices comprise infrared reflective material, arranged to be illuminated by an infrared light source.

8. Apparatus according to any preceding claim, wherein the or each game controller device is uniquely identifiable and continuously transmits positional data relative to the projection surface.

9. Apparatus according to any preceding claim, and further comprising a central data processing device serving to capture and relay operational data for the or each game controller.

10. Apparatus according to any preceding claim, wherein the calibration device is an automatic-calibration device including its own separate data processing device.

11. Apparatus according to any preceding claim, wherein the further electromagnetic sensing device serves to detect electromagnetic radiation in both the visible and invisible spectrum.

12. A method of tracking at least one computer game controller device relative to an array of marker devices in a detection space, the method comprising:• displaying an image on a projection screen, the projection screen including the array of marker devices,• detecting electromagnetic radiation in the invisible spectrum enabling detection of the array of markers and processing data relating thereto,• utilising the data from the electromagnetic radiation in the invisible spectrum and detection of the visible image to calculate precise location data of each marker in the array of markers relative to the detection space, and• communicating the precise location data to the at least one computer game controller device to calculate its own position relative the precise location of each marker in the array in the detection space.

13. A method according to claim 12, wherein the or each controller device calculates its position relative to the projection surface.

14. A method according to claim 12 or 13, wherein infrared illumination of the array of markers enables the array of markers to be detectable by an electromagnetic sensing device of the or each game controller.

15. A method according to any one of claims 12 to 14, wherein the or each game controller is uniquely identifiable and continuously transmits positional data relative to the projection surface.

16. A method according to any one of claims 12 to 15, and further comprising a dedicated communication protocol between the or each game controller and acentral data processing device serving to capture and relay operational data for the or each game controller.

17. A method according to any one of claims 12 to 16, wherein the calibration device serves to calculate offset data of the array of markers relative to the projected image on the projection surface, enabling compensating for variance in 3D / real world positioning changes or image projector device skew and / or manufacturing / construction variances.

18. A method according to any one of claims 14 to 17, and comprising one or more of the following:■ image acquisition whereby the electromagnetic sensing device of the game controller device captures images of the detection space,■ pre-processing wherein a long pass filter of the game controller device is used to reduce the amount of extraneous information visible to the electromagnetic sensing device,■ marker detection in which images are analysed by a detection algorithm to identify the markers,■ position calculation including detecting a specific pattern of the markers, the detected specific pattern used to calculate the position and orientation of the game controller device relative to the markers,■ real-time feedback.

Images