Information processing system and information processing method

Abstract

[Problem] To accurately correct positional errors in inertial motion capture. [Solution] Provided is an information processing system comprising: an acquisition unit that acquires inertial data obtained by each of a plurality of inertial motion capture devices attached respectively to a plurality of sites on a user, including a reference site, and position measurement data obtained using a wireless tag attached to the reference site and synchronized with the inertial data; and a correction unit that corrects, on the basis of the position measurement data, the pose of the reference site estimated on the basis of the inertial data, wherein the correction unit estimates the poses of a plurality of joints, including the reference site, on the basis of the corrected pose of the reference site.

Description

Information Processing System and Information Processing Method

[0001] The present disclosure relates to an information processing system and an information processing method.

[0002] In recent years, technologies for capturing a user's motion using sensors have been developed. Also, technologies for improving the accuracy of capture have been developed. For example, Patent Document 1 discloses a technique for supplementing the position of a missing optical marker by an inertial method based on optical motion capture.

[0003] Japanese Unexamined Patent Application Publication No. 2023 - 024042

[0004] Unlike Patent Document 1, when based on inertial motion capture, the accumulation of errors derived from an IMU (Inertial Measurement Unit) becomes an issue.

[0005] According to an aspect of the present disclosure, there is provided an information processing system including: an acquisition unit that acquires inertial data acquired by each of a plurality of inertial motion capture devices attached to each of a plurality of parts including a reference part of a user, and positioning data acquired using a wireless tag attached to the reference part and synchronized with the inertial data; and a correction unit that corrects the pose of the reference part estimated based on the inertial data based on the positioning data, wherein the correction unit estimates the poses of a plurality of joints including the reference part based on the corrected pose of the reference part.

[0006] Also, according to another aspect of the present disclosure, there is provided an information processing method including: the processor acquiring inertial data acquired by each of a plurality of inertial motion capture devices attached to each of a plurality of parts including a reference part of the user, and positioning data acquired using a wireless tag attached to the reference part and synchronized with the inertial data; correcting the pose of the reference part estimated based on the inertial data based on the positioning data; and estimating the poses of a plurality of joints including the reference part based on the corrected pose of the reference part.

[0007] This is a block diagram showing an example of the functional configuration of an information processing system 1 according to one embodiment of the present disclosure. This is a diagram showing an example of a measurement environment according to the same embodiment. This is a block diagram showing an example of the functional configuration of an information processing device 30 according to the same embodiment. This is a flowchart illustrating an example of the processing flow by the preprocessing unit 320 according to the same embodiment. This is a diagram illustrating an example of the processing flow by the estimation unit 350 according to the same embodiment. This is a flowchart illustrating an example of the correction flow based on the magnitude of motion according to the same embodiment. This is a diagram showing an example of the determination of intensity parameters according to the same embodiment. This is a diagram showing an example of the determination of intensity parameters according to the same embodiment. This is a block diagram showing an example of the hardware configuration of an information processing device 90 according to the same embodiment.

[0008] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0009] Furthermore, in this specification and drawings, when describing multiple identical components to distinguish them, letters or other symbols may be added to the end of the reference numerals. On the other hand, when there is no need to distinguish multiple identical components, the letters or other symbols may be omitted, and a description common to all identical components may be provided.

[0010] The explanation will be presented in the following order: 1. Embodiments 1.1. Overview 1.2. Example Functional Configuration 1.3. Detailed Functionality 2. Example Hardware Configuration 3. Summary

[0011] <1. Embodiments> <<1.1. Overview>> In inertial motion capture, the motion of the user's entire body is estimated based on acceleration and angular velocity (hereinafter, both are collectively referred to as inertial data) acquired by multiple IMUs attached to the user's body.

[0012] Inertial motion capture has the advantage of being usable both indoors and outdoors because it does not require external equipment such as cameras.

[0013] On the other hand, movement measurement using an IMU is achieved by accumulating past movement data. Therefore, there is a problem in that positional errors also accumulate due to the accumulation of noise, bias, etc., over time.

[0014] For example, when using captured motion data in VR (Virtual Reality), AR (Augmented Reality), etc., the accumulated positional differences described above may cause users to experience visual discomfort or make interaction with other users difficult.

[0015] Furthermore, for example, when simultaneously capturing the motion of multiple users in the same space, positional errors may occur in each user's motion, potentially leading to situations where the motions overlap.

[0016] The technical concept of one embodiment of this disclosure was conceived with the above-mentioned points in mind, and enables highly accurate correction of positional errors in inertial motion capture.

[0017] To this end, one of the features of an information processing method according to one embodiment of the present disclosure is that it combines inertial motion capture with a positioning method that enables positioning in space with a predetermined accuracy regardless of the passage of time.

[0018] The following will describe in detail an example of the functional configuration of an information processing system 1 that implements the information processing method having the above-described features, with reference to Figures 1 to 3.

[0019] <<1.2. Example of Functional Configuration>> Figure 1 is a block diagram showing an example of the functional configuration of an information processing system 1 according to one embodiment of the present disclosure. Figure 2 is a diagram showing an example of a measurement environment according to the present embodiment.

[0020] As shown in Figure 1, the information processing system 1 according to this embodiment includes an inertial motion capture device 10, a positioning system 20, and an information processing device 30.

[0021] (Inertial motion capture device 10) The inertial motion capture device 10 according to this embodiment is a device that realizes inertial motion capture.

[0022] Multiple inertial motion capture devices 10 according to this embodiment are attached to predetermined body parts of the user.

[0023] In the example shown in Figure 2, the specified body parts include the waist, head, right wrist, left wrist, right ankle, and left ankle, and each body part is fitted with an inertial motion capture device 10A to 10F.

[0024] Each of the inertial motion capture devices 10A to 10F is equipped with an IMU and acquires inertial data corresponding to the movement of the corresponding body part.

[0025] Furthermore, each of the inertial motion capture devices 10A to 10F is equipped with a communication device for transmitting the acquired inertial data.

[0026] Furthermore, one of the features of the inertial motion capture devices 10, specifically the one attached to the reference body part, is that it is equipped with a wireless tag that transmits and receives wireless signals to and from the positioning system 20.

[0027] The above-mentioned reference area may be any part of the body that can be used as a reference for the user's movements or position. In this embodiment, the reference area is the waist, and the inertial motion capture device 10A attached to the waist is equipped with the above-mentioned wireless tag.

[0028] However, the reference area may be the buttocks, abdomen, groin, etc.

[0029] (Positioning System 20) The positioning system 20 according to this embodiment is a system that performs spatial positioning with a predetermined accuracy regardless of the passage of time.

[0030] The positioning system 20 according to this embodiment may perform positioning using, for example, wireless signals.

[0031] The positioning system 20 according to this embodiment includes a plurality of wireless anchors 25 that are arranged at predetermined positions in space 50.

[0032] Multiple wireless anchors 25 are devices used to determine the position of a wireless tag in space 50, and each of them transmits and receives wireless signals to and from the wireless tag.

[0033] In the example shown in Figure 2, the positioning system 20 includes seven wireless anchors 25A to 25F and 25M.

[0034] Wireless anchor 25M is the master that controls the operation of the other wireless anchors 25A to 25F. Wireless anchors 25A to 25F are slaves that operate according to the control of wireless anchor 25M. Wireless anchor 25M transmits and receives signals related to operation control between itself and each of the wireless anchors 25A to 25F.

[0035] Furthermore, the positioning system 20 according to this embodiment includes a receiver 27 that receives inertial data and positioning data.

[0036] The receiver 27 transmits the received inertial data and positioning data to the information processing device 30.

[0037] For example, the receiver 27 may receive inertial data acquired by each of the inertial motion capture devices 10.

[0038] On the other hand, the receiver 27 may receive inertial data acquired by each of the inertial motion capture devices 10A to 10F from the inertial motion capture device 10A attached to the reference area.

[0039] Furthermore, for example, the receiver 27 may receive positioning data from an inertial motion capture device 10A attached to a reference area.

[0040] In this case, the inertial motion capture device 10A may transmit inertial data and positioning data on the same wireless signal.

[0041] On the other hand, the receiver 27 may receive positioning data from the wireless anchor 25M operating as a master.

[0042] The positioning data according to the present embodiment may be information that is acquired in synchronization with inertial data and is used for positioning the position of the wireless tag within the space 50.

[0043] Therefore, the positioning data according to the present embodiment includes information regarding the distance between each of three or more wireless anchors 25 and the wireless tag.

[0044] The distance between the wireless anchor 25 and the wireless tag can be calculated based on, for example, the time when one of the wireless anchor 25 or the wireless tag transmits a wireless signal, the time when the other receives the wireless signal, and the propagation speed of the wireless signal.

[0045] Further, if the distance between each of three or more wireless anchors 25 and the wireless tag can be calculated and the position of each of the three or more wireless anchors 25 in the space 50 is known, the position of the wireless tag in the space 50 can be calculated.

[0046] The format of the positioning data according to the present embodiment is not limited as long as it is information sufficient for calculating the position of the wireless tag within the space 50.

[0047] The positioning data may include, for example, information on the distance between the wireless anchor 25 and the wireless tag, or may include information on the transmission time and reception time for calculating the distance.

[0048] Also, the transmission direction of the wireless signal between each of the wireless anchors 25 and the wireless tag is not limited. Each of the wireless anchors 25 may receive a wireless signal transmitted by the wireless tag. On the other hand, the wireless tag may receive wireless signals transmitted simultaneously by each of the wireless anchors 25. On the other hand, each of the wireless anchors 25 and the wireless tag may perform two-way communication.

[0049] Furthermore, the radio signals transmitted and received between each of the radio anchors 25 and the radio tag may be UWB (Ultra-wide band) signals. That is, the radio anchors 25 may be UWB anchors, and the radio tags may be UWB tags.

[0050] By using radio waves with extremely short pulse widths of less than nanoseconds, such as UWB signals, the propagation time of radio waves can be measured with high accuracy, enabling high-precision distance measurement based on propagation time.

[0051] However, the radio signals transmitted and received between each of the wireless anchors 25 and the wireless tag are not limited to the above example.

[0052] The positioning system 20 according to this embodiment may employ other communication standards such as Wi-Fi®, Bluetooth® Low Energy, etc.

[0053] Furthermore, the above description explains the case in which the receiver 27 transmits the received inertial data and positioning data to the information processing device 30, but this is merely one example.

[0054] Inertial data and positioning data can be synchronized using, for example, a timestamp, and finally acquired by the information processing device 30. The transmission path and transmission protocol can be designed as appropriate.

[0055] Next, we will describe an example of the functional configuration of the information processing device 30 according to this embodiment. Figure 3 is a block diagram showing an example of the functional configuration of the information processing device 30 according to this embodiment.

[0056] As shown in Figure 3, the information processing device 30 according to this embodiment may include a communication unit 310, a preprocessing unit 320, a positioning result storage unit 330, an inertial data storage unit 340, an estimation unit 350, and an estimation result storage unit 360, etc.

[0057] (Communication Unit 310) The communication unit 310 according to this embodiment operates as an acquisition unit that acquires inertial data acquired by each of the multiple inertial motion capture devices 10 attached to each of the multiple body parts including the user's reference body part, and positioning data acquired using a wireless tag attached to the reference body part and synchronized with the inertial data.

[0058] For example, the communication unit 310 receives inertial data and positioning data from the receiver 27.

[0059] (Pre-processing unit 320) The pre-processing unit 320 according to this embodiment performs pre-processing such as positioning of the wireless tag based on positioning data.

[0060] The functions of the preprocessor 320 according to this embodiment are realized through the cooperation of various processors and memory. The functions of the preprocessor 320 according to this embodiment will be described later.

[0061] (Position positioning result storage unit 330) The position positioning result storage unit 330 according to this embodiment stores the position positioning result obtained by the preprocessing unit 320.

[0062] (Inertial data storage unit 340) The inertial data storage unit 340 according to this embodiment stores inertial data in a predetermined format.

[0063] (Estimation Unit 350) The estimation unit 350 according to this embodiment performs user pose estimation based on inertial data. The time-series pose estimation result by the estimation unit 350 can be said to be data in which the user's motion has been captured (motion capture data).

[0064] Furthermore, the estimation unit 350 according to this embodiment operates as a correction unit that corrects the pose estimated based on inertial data based on positioning data.

[0065] For example, the estimation unit 350 according to this embodiment may correct the pose of a reference part estimated based on inertial data based on positioning data, and estimate the poses of a plurality of joints including the reference part based on the corrected pose of the reference part.

[0066] The functions of the estimation unit 350 according to this embodiment are realized through the cooperation of various processors and memory. The functions of the estimation unit 350 according to this embodiment will be described later.

[0067] (Estimated result storage unit 360) The estimated result storage unit 360 according to this embodiment stores the estimated result from the estimation unit 350.

[0068] The above describes an example of the functional configuration of the information processing system 1 according to this embodiment. However, the above functional configuration described with reference to Figures 1 to 3 is merely an example, and the functional configuration example of the information processing device 1 according to this embodiment is not limited to this example.

[0069] For example, the configuration described above for the information processing device 30 does not necessarily have to be provided in a single device.

[0070] Furthermore, for example, the above example illustrates a case where the inertial motion capture device 10 attached to the reference area is equipped with a wireless tag, but the wireless tag may be attached to the reference area separately from the inertial motion capture device 10.

[0071] Furthermore, while the above example illustrates the case of determining the position of a reference part based on a wireless signal, the position of the reference part may also be determined by, for example, SLAM (Simultaneous Localization and Mapping).

[0072] In this case, the positioning system 20 may be a wearable device such as an HMD (Head Mounted Display).

[0073] The functional configuration of the information processing system 1 according to this embodiment can be flexibly modified according to specifications, operation, etc.

[0074] <<1.3. Functional Details>> Next, the functions of the information processing device 30 according to this embodiment will be described in detail.

[0075] First, the functions of the preprocessing unit 320 according to this embodiment will be described. Figure 4 is a flowchart illustrating an example of the processing flow by the preprocessing unit 320 according to this embodiment.

[0076] In the example shown in Figure 4, first, the communication unit 310 receives positioning data and inertial data (S101).

[0077] Next, the preprocessor 320 separates the positioning data and inertial data received by the communication unit 310 in step S101 (S102).

[0078] If the communication unit 310 receives positioning data and inertial data via a separate wireless signal in step S101, step S102 may be skipped.

[0079] The pre-processing unit 320 performs positioning of the reference part based on positioning data (S103), and stores the positioning result in the positioning result storage unit 330 (S104).

[0080] The preprocessing unit 320, for example, associates the timestamp with the positioning result (x, y) of the reference part and stores it in the positioning result storage unit 330.

[0081] Furthermore, the preprocessing unit 320 converts the inertial data into an estimation format (S105) and stores the converted inertial data in the inertial data storage unit 340 (S106).

[0082] The above estimation format may be, for example, a format in which a timestamp is associated with the acceleration and angular velocity of the six axes.

[0083] Next, the functions of the estimation unit 350 according to this embodiment will be described in detail. Figure 5 is a diagram illustrating an example of the processing flow by the estimation unit 350 according to this embodiment.

[0084] First, we will describe the process when no correction is performed based on the positioning results according to this embodiment, i.e., the conventional processing flow.

[0085] First, the estimation unit 350 calculates the position and orientation (position and orientation together are simply referred to as pose) of the joints corresponding to each mounting part of the inertial motion capture device 10 using inertial navigation based on the inertial data.

[0086] Specifically, the estimation unit 350 calculates global acceleration and attitude based on inertial data (S201), and performs integral calculations based on global acceleration and attitude (S202). The joint poses corresponding to each attachment site, obtained by the integral calculation in step S202, are input to the Kalman filter.

[0087] Furthermore, the estimation unit 350 calculates the confidence level and the pose of the joints corresponding to each attachment site based on the motion model.

[0088] Specifically, the estimation unit 350 selects a motion model (e.g., a whole-body model, an upper-limb model, etc.) according to the attachment site (S203), and performs joint position regression processing with motion constraints according to the motion model to estimate the pose of each joint (S204). The estimation unit 350 also calculates the confidence level of the estimation and performs extraction based on the confidence level (S205). The extracted joint poses corresponding to each attachment site are input to the Kalman filter.

[0089] In this way, by performing extraction based on confidence levels, it is possible to dynamically change the adoption rate of poses calculated based on the motion model (i.e., the merge rate with poses calculated by inertial navigation).

[0090] If no correction is performed based on the positioning results according to this embodiment, the estimation unit 350 then uses a Kalman filter to perform a correction calculation based on the pose calculated by inertial navigation and the pose calculated based on the motion model (S207).

[0091] Next, the estimation unit 350 interpolates the joint positions of the entire body based on the IK (Inverse Kinematics) model, using the results of the calculation in step S207, i.e., the poses of the joints corresponding to each corrected attachment site (for example, 6 joints), and calculates (estimates) the poses of all the joints in the entire body (for example, 27 joints) (S208).

[0092] The estimation unit 350 stores the estimated joint pose result 60 of the whole body, obtained by the calculation in step S208, in the estimation result storage unit 360.

[0093] The estimation unit 350 repeatedly performs the above-described process frame by frame based on the timestamp to obtain a time-series whole-body joint pose estimation result 60.

[0094] Next, we will describe the processing flow when performing corrections based on the positioning results according to this embodiment.

[0095] As described above, the positioning results according to this embodiment are obtained by a positioning method that enables positioning in space with a predetermined accuracy regardless of the passage of time. The positioning results obtained by such a method make it possible to correct position errors in inertial motion capture with high accuracy.

[0096] However, if the positioning results are directly reflected in the whole-body joint pose estimation results 60, the feet of the model to which the motion is applied may appear to slide sideways (position drift), potentially reducing the sense of the model's contact with the ground.

[0097] Therefore, the estimation unit 350 according to this embodiment may correct the pose of the reference part estimated based on inertial data by inputting the position of the reference part (positioning result) calculated based on positioning data to the Kalman filter (S206).

[0098] In this case, in step S207, the estimation unit 350 uses a Kalman filter to perform correction calculations based on the pose calculated by inertial navigation, the pose calculated based on the motion model, and the positioning results of the reference part.

[0099] Furthermore, in the subsequent step S208, the estimation unit 350 interpolates the joint positions of the entire body based on the IK model from the poses of the joints corresponding to each corrected attachment site, and calculates (estimates) the poses of the joints of the entire body.

[0100] In other words, the estimation unit 350 estimates the poses of all joints in the body based on the corrected reference body pose and the IK model.

[0101] In this way, by performing ground contact processing using an IK model after correcting the pose of the reference part using the positioning results, it is possible to reduce the aforementioned position drift and ensure a sense of contact between the model and the ground.

[0102] Next, we will describe how the intensity of the correction according to this embodiment is determined. As described above, the estimation unit 350 according to this embodiment corrects the pose of the whole body based on the positioning result of the reference part.

[0103] However, if the above corrections are applied to a stationary model, the positional drift mentioned above will be exaggerated, further reducing the sense of the model's connection to the ground.

[0104] Therefore, the estimation unit 350 according to this embodiment may reduce positional drift and ensure a sense of contact between the model and the ground by determining the strength of the correction based on the magnitude of the motion.

[0105] Figure 6 is a flowchart showing an example of the correction flow based on the magnitude of motion according to this embodiment.

[0106] In the example shown in Figure 6, the estimation unit 350 first obtains the inertia data for the relevant frame from the inertia data storage unit 340 (S301).

[0107] Next, the estimation unit 350 retrieves the positioning result from the positioning result storage unit 330 based on the timestamp of the inertial data acquired in step S301 (S302).

[0108] For example, the estimation unit 350 may search for positioning results associated with timestamps where the difference from the timestamp of the inertial data is less than n seconds.

[0109] If a suitable positioning result is obtained (S303: YES), the estimation unit 350 determines an intensity parameter that specifies the correction intensity based on the magnitude of the motion (S304).

[0110] In this case, the estimation unit 350 may estimate the magnitude of the motion based on at least one of inertial data or positioning data.

[0111] More specifically, the estimation unit 350 may estimate the magnitude of the motion based on the difference in position between the current frame (the relevant frame) and previous frames.

[0112] The above location may be the location of the reference area. Alternatively, the above location may be the location of the ankle joint.

[0113] The estimation unit 350 inputs the positioning result to the Kalman filter based on the intensity parameter determined in step S304 (S305).

[0114] Subsequently, the estimation unit 350 specifies the next frame (S306) and returns to step S301.

[0115] If a suitable positioning result cannot be obtained (S303: NO), the estimation unit 350 skips steps S304 and S305.

[0116] Next, with reference to Figures 7 to 9, an example of determining the intensity parameters according to this embodiment will be described. Figures 7 to 9 are diagrams showing examples of determining the intensity parameters according to this embodiment.

[0117] In the example shown in Figure 7, the estimation unit 350 may set the intensity parameter to maximum if the difference in position between the current frame and previous frames exceeds a threshold, and to minimum (0) the intensity parameter if the difference is less than or equal to the threshold.

[0118] In other words, the estimation unit 350 may perform a correction based on the positioning result only if the estimated magnitude of the motion exceeds a threshold.

[0119] On the other hand, the estimation unit 350 may change the strength of the correction according to the magnitude of the motion.

[0120] In the example shown in Figure 8, the estimation unit 350 determines the intensity parameter using a proportionality constant. In the example shown in Figure 9, the estimation unit 350 determines the intensity parameter using an exponential function.

[0121] By determining the intensity parameters as illustrated in Figures 7 to 9, it becomes possible to further reduce positional drift and ensure a sense of contact between the model and the ground.

[0122] The processing flow by the estimation unit 350 according to this embodiment has been described above. The collection of inertial data and positioning data, the estimation of the pose of the reference part based on the inertial data, and the correction of the pose of the reference part based on the positioning data described above may be performed in real time.

[0123] According to the information processing system 1 of this embodiment, for example, in live broadcasts using motion capture, it is possible to correct positional errors in inertial motion capture with high accuracy.

[0124] On the other hand, pose estimation of the reference part based on inertial data, and correction of the reference part's pose based on positional data, may be performed at any time after the collection of inertial data and positional data.

[0125] In this case, the estimation unit 350 may determine the intensity parameters by referring to frames from the current frame onward.

[0126] For example, in a situation where the character is transitioning from a walking state to a stationary state, if only past frames are referenced, the correction intensity may become too strong immediately after the switch to the stationary state (potential positional drift may occur). However, by referencing frames from the current frame onward, it becomes possible to determine intensity parameters that take into account the future transition to the stationary state, thereby preventing positional drift.

[0127] Furthermore, the collection of inertial data and positioning data, the estimation of the reference part's pose based on the inertial data, and the correction of the reference part's pose based on the positioning data may be performed for each of the multiple users.

[0128] In this case, the inertial data and positioning data for each of the multiple users may be collected in the same space.

[0129] On the other hand, inertial data and positioning data for each of multiple users may be collected in separate spaces. In this case, if the coordinates of the spaces are synchronized, it becomes possible to accurately reproduce the interaction between users in different spaces using motion data.

[0130] <2. Hardware Configuration Example> Next, a hardware configuration example of the information processing device 90 according to one embodiment of the present disclosure will be described. Figure 10 is a block diagram showing a hardware configuration example of the information processing device 90 according to one embodiment of the present disclosure.

[0131] The information processing device 90 may be a device having the same hardware configuration as the information processing device 30. As shown in Figure 10, the information processing device 90 may include, for example, a processor 871, a ROM 872, a RAM 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, an input device 878, an output device 879, a storage device 880, a drive 881, a connection port 882, and a communication device 883. Note that the hardware configuration shown here is just an example, and some of the components may be omitted. Furthermore, the information processing device 90 may include components other than those shown here.

[0132] (Processor 871) The processor 871 functions, for example, as an arithmetic processing unit or a control unit, and controls the overall operation or part thereof of each component based on various programs recorded in the ROM 872, RAM 873, storage 880, or removable storage medium 901.

[0133] (ROM 872, RAM 873) ROM 872 is a means for storing programs loaded into the processor 871 and data used for calculations. RAM 873 temporarily or permanently stores, for example, programs loaded into the processor 871 and various parameters that change as needed when executing those programs.

[0134] (Host bus 874, bridge 875, external bus 876, interface 877) The processor 871, ROM 872, and RAM 873 are interconnected via, for example, the host bus 874, which is capable of high-speed data transmission. On the other hand, the host bus 874 is connected to the external bus 876, which has a relatively low data transmission speed, via, for example, the bridge 875. The external bus 876 is also connected to various components via the interface 877.

[0135] (Input device 878) The input device 878 may include, for example, a mouse, keyboard, touch panel, buttons, switches, and levers. Furthermore, the input device 878 may also include a remote controller (hereinafter referred to as a remote control) capable of transmitting control signals using infrared rays or other radio waves. The input device 878 may also include an audio input device such as a microphone.

[0136] (Output device 879) The output device 879 is a device that can visually or audibly notify the user of acquired information, such as a display device such as a CRT (Cathode Ray Tube), LCD, or organic EL, an audio output device such as a speaker or headphones, a printer, a mobile phone, or a facsimile. The output device 879 according to this disclosure also includes various vibration devices capable of outputting tactile stimuli.

[0137] (Storage 880) Storage 880 is a device for storing various types of data. Examples of storage devices used for storage 880 include magnetic storage devices such as hard disk drives (HDDs), semiconductor storage devices, optical storage devices, or magneto-optical storage devices.

[0138] (Drive 881) Drive 881 is a device that reads information recorded on a removable storage medium 901, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, or writes information to the removable storage medium 901.

[0139] (Removable storage medium 901) The removable storage medium 901 is, for example, DVD media, Blu-ray® media, HD DVD media, various semiconductor storage media, etc. Of course, the removable storage medium 901 may also be, for example, an IC card equipped with a contactless IC chip, or an electronic device, etc.

[0140] (Connection port 882) Connection port 882 is a port for connecting external devices 902, such as a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or an optical audio terminal.

[0141] (External connected device 902) The external connected device 902 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

[0142] (Communication device 883) The communication device 883 is a communication device for connecting to a network, and is, for example, a communication card for wired or wireless LAN, Bluetooth®, or WUSB (Wireless USB), a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

[0143] <3. Summary> As described above, an information processing system according to one embodiment of the present disclosure includes: an acquisition unit that acquires inertial data acquired by each of a plurality of inertial motion capture devices attached to each of a plurality of parts including a reference part of the user, and positioning data acquired using a wireless tag attached to the reference part and synchronized with the inertial data; and a correction unit that corrects the pose of the reference part estimated based on the inertial data based on the positioning data. Furthermore, one of the features of the correction unit is that it estimates the poses of a plurality of joints including the reference part based on the corrected pose of the reference part.

[0144] The above configuration makes it possible to correct positional errors in inertial motion capture with high precision.

[0145] While preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art of the present disclosure that various modifications or alterations may be conceived within the scope of the technical ideas described in the claims, and these will naturally also fall within the technical scope of the present disclosure.

[0146] Furthermore, each step of the process described in this disclosure does not necessarily have to be processed chronologically in the order shown in the flowchart or sequence diagram. For example, each step of the process for each device may be processed in an order different from the order described, or may be processed in parallel.

[0147] Furthermore, the series of processes performed by each device described in this disclosure may be implemented by a program stored on a non-transitory computer-readable storage medium. Each program is, for example, loaded into RAM when executed by a computer and executed by a processor such as a CPU. The storage medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, or flash memory. Alternatively, the program may be distributed without using a storage medium, for example, via a network.

[0148] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that are obvious to those skilled in the art from the description herein, in addition to or instead of the effects described herein.

[0149] Furthermore, the following configurations also fall within the technical scope of this disclosure: (1) An information processing system comprising: an acquisition unit that acquires inertial data acquired by each of a plurality of inertial motion capture devices attached to each of a plurality of parts including a reference part of the user, and positioning data acquired using a wireless tag attached to the reference part and synchronized with the inertial data; and a correction unit that corrects the pose of the reference part estimated based on the inertial data based on the positioning data, wherein the correction unit estimates the poses of a plurality of joints including the reference part based on the corrected pose of the reference part. (2) The information processing system according to (1), wherein the inertial motion capture device attached to the reference part comprises the wireless tag. (3) The information processing system according to either (1) or (2), wherein the wireless tag is a UWB tag. (4) The information processing system according to (3), wherein the positioning data includes information regarding the distance between each of three or more UWB anchors and the UWB tag. (5) The information processing system according to any one of (1) to (4), wherein the correction unit corrects the pose of the reference part estimated based on the inertial data based on the position of the reference part calculated based on the positioning data. (6) The information processing system according to (5), wherein the correction unit corrects the pose of the reference part estimated based on the inertial data by inputting the position of the reference part calculated based on the positioning data into a Kalman filter. (7) The information processing system according to any one of (1) to (6), wherein the correction unit estimates the pose of all joints of the whole body based on the corrected pose of the reference part and an IK model. (8) The information processing system according to any one of (1) to (7), wherein the correction unit determines the strength of the correction based on the magnitude of the motion. (9) The information processing system according to (8), wherein the correction unit estimates the magnitude of the motion based on at least one of the inertial data or the positioning data.(10) The information processing system according to either (8) or (9), wherein the correction unit performs correction when the magnitude of motion exceeds a threshold. (11) The information processing system according to either (8) or (9), wherein the correction unit changes the strength of the correction according to the magnitude of motion. (12) The information processing system according to any one of (1) to (11), wherein the collection of inertial data and positioning data, pose estimation of the reference part based on the inertial data, and correction of the pose of the reference part based on positioning data by the correction unit are performed in real time. (13) The information processing system according to any one of (1) to (12), wherein the collection of inertial data and positioning data, pose estimation of the reference part based on the inertial data, and correction of the pose of the reference part based on positioning data by the correction unit are performed for each of the multiple users. (14) The information processing system according to any one of (1) to (13), wherein the reference part is the waist. (15) The information processing system according to any one of (1) to (14), further comprising a plurality of the inertial motion capture devices. (16) The information processing system according to any one of (1) to (15), further comprising the wireless tag. (17) The information processing system according to any one of (1) to (16), further comprising three or more wireless anchors that transmit and receive wireless signals between themselves and the wireless tag. (18) An information processing method comprising: a processor acquiring inertial data acquired by each of a plurality of inertial motion capture devices attached to each of a plurality of parts including a reference part of the user, and positioning data acquired using a wireless tag attached to the reference part and synchronized with the inertial data; correcting the pose of the reference part estimated based on the inertial data based on the positioning data; and estimating the poses of a plurality of joints including the reference part based on the corrected pose of the reference part.

[0150] 1. Information processing system 10. Inertial motion capture device 20. Positioning system 25. Wireless anchor 27. Receiver 30. Information processing device 310. Communication unit 320. Preprocessing unit 330. Positioning result storage unit 340. Inertial data storage unit 350. Estimation unit 360. Estimation result storage unit

Claims

1. An information processing system comprising: an acquisition unit that acquires inertial data acquired by each of a plurality of inertial motion capture devices attached to each of a plurality of parts including a reference part of the user, and positioning data acquired using a wireless tag attached to the reference part and synchronized with the inertial data; and a correction unit that corrects the pose of the reference part estimated based on the inertial data based on the positioning data, wherein the correction unit estimates the poses of a plurality of joints including the reference part based on the corrected pose of the reference part.

2. The information processing system according to claim 1, wherein the inertial motion capture device attached to the reference part comprises the wireless tag.

3. The information processing system according to claim 1, wherein the wireless tag is a UWB tag.

4. The information processing system according to claim 3, wherein the positioning data includes information relating to the distance between each of three or more UWB anchors and the UWB tag.

5. The information processing system according to claim 1, wherein the correction unit corrects the pose of the reference part estimated based on the inertial data based on the position of the reference part calculated based on the positioning data.

6. The information processing system according to claim 5, wherein the correction unit corrects the pose of the reference part estimated based on the inertial data by inputting the position of the reference part calculated based on the positioning data to a Kalman filter.

7. The information processing system according to claim 1, wherein the correction unit estimates the poses of all joints of the body based on the corrected pose of the reference part and the IK model.

8. The information processing system according to claim 1, wherein the correction unit determines the strength of the correction based on the magnitude of the motion.

9. The information processing system according to claim 8, wherein the correction unit estimates the magnitude of motion based on at least one of the inertial data or the positioning data.

10. The information processing system according to claim 8, wherein the correction unit performs a correction when the magnitude of the motion exceeds a threshold.

11. The information processing system according to claim 8, wherein the correction unit changes the strength of the correction according to the magnitude of the motion.

12. The information processing system according to claim 1, wherein the collection of inertial data and positioning data, the estimation of the pose of the reference part based on the inertial data, and the correction of the pose of the reference part based on the positioning data by the correction unit are performed in real time.

13. The information processing system according to claim 1, wherein the collection of inertial data and positioning data, the estimation of the pose of the reference part based on the inertial data, and the correction of the pose of the reference part based on the positioning data by the correction unit are performed for each of the multiple users.

14. The information processing system according to claim 1, wherein the reference part is the lumbar region.

15. The information processing system according to claim 1, further comprising a plurality of the inertial motion capture devices.

16. The information processing system according to claim 1, further comprising the wireless tag.

17. The information processing system according to claim 1, further comprising three or more wireless anchors that transmit and receive wireless signals between themselves and the wireless tag.

18. An information processing method comprising: a processor acquiring inertial data acquired by each of a plurality of inertial motion capture devices attached to each of a plurality of body parts including a reference body part of a user; positioning data acquired using a wireless tag attached to the reference body part and synchronized with the inertial data; correcting the pose of the reference body part estimated based on the inertial data based on the positioning data; and estimating the poses of a plurality of joints including the reference body part based on the corrected pose of the reference body part.

Images