Information processing device and information processing method

Abstract

An information processing device according to the present invention is a device that is worn by a user, and comprises: a processor that controls reproduction of a video virtual object; and a memory that stores control information which indicates control with respect to a prescribed object included in the video virtual object. When the processor reproduces the video virtual object and determines that the user has performed a first operation, the processor performs first control with respect to the prescribed object on the basis of the control information and continues reproduction of the video virtual object other than the prescribed object. When the processor determines that the user has performed a second operation, the processor performs second control with respect to the prescribed object on the basis of the control information which has been stored in the memory.

Description

Information processing apparatus and information processing method

[0001] The present invention relates to an information processing apparatus and an information processing method.

[0002] Patent Document 1 discloses an information processing apparatus and an information processing method that are worn and used by a user. In this information processing apparatus and information processing method, a three-dimensional video virtual object is reproduced, and the reproduced video virtual object is arranged and displayed so as to be superimposed on a real object, and the arrangement position where the video virtual object is arranged and the user's position and at least one of the line-of-sight directions are recorded. Also, when it is detected that the user has moved outside a range that is a predetermined distance from the arrangement position, the reproduction of the video virtual object is stopped. After the reproduction of the video virtual object is stopped, when the user enters the above range, the reproduction of the video virtual object is restarted based on the recorded user information.

[0003] International Publication No. WO2023 / 233488

[0004] However, Patent Document 1 does not disclose classifying three-dimensional video virtual objects and performing control of stop and reproduction, etc., and there is room for improvement in enabling the user to view three-dimensional video virtual objects more suitably.

[0005] According to a first aspect of the present invention, the following information processing apparatus is provided. This information processing apparatus is a device worn by a user, and includes a processor that controls the reproduction of a video virtual object, and a memory that stores control information indicating control of a predetermined object included in the video virtual object. When the processor determines that the user has performed a first operation, the processor performs first control on the predetermined object based on the control information stored in the memory, and continues the reproduction of video virtual objects other than the predetermined object. Then, when the processor determines that the user has performed a second operation, the processor performs second control on the predetermined object based on the control information stored in the memory.

[0006] According to a second aspect of the present invention, the following information processing method is provided. This information processing method is used in an information processing device and stores control information indicating control over a predetermined object included in a video virtual object, and plays back the video virtual object. When this method determines that the user of the information processing device has performed a first action, it performs a first control over the predetermined object based on the stored control information and continues playing back video virtual objects other than the predetermined object. Then, when this method determines that the user has performed a second action, it performs a second control over the predetermined object based on the stored control information.

[0007] The present invention provides a technology that enables more favorable viewing of virtual video objects. Other problems, configurations, and effects not mentioned above will be clarified by the following description of embodiments for carrying out the invention.

[0008] This is a diagram showing an example of an information processing device. This is a diagram illustrating an example of the usage of an information processing device. This is a diagram showing an example of the configuration of an information processing device. This is a diagram illustrating an example of the functional configuration of an information processing device. This is a diagram showing an example of viewing assistance control information. This is a flowchart showing an example of processing by an information processing device. This is a diagram illustrating an example of viewing assistance control. This is a diagram illustrating an example of viewing assistance control. This is a diagram illustrating an example of viewing assistance control. This is a flowchart showing an example of processing by an information processing device. This is a flowchart showing an example of processing by an information processing device. This is a diagram illustrating an example of viewing assistance control. This is a diagram illustrating an example of viewing assistance control.

[0009] Embodiments of the present invention will be described below with reference to the drawings. The embodiments are illustrative examples for explaining the present invention, and have been omitted and simplified as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0010] The positions, sizes, shapes, and ranges of the components shown in the drawings may not represent their actual positions, sizes, shapes, and ranges in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, and ranges disclosed in the drawings.

[0011] Examples of various types of information may be described using terms such as "table," "list," "queue," and "file," but these types of information may also be represented by other data structures. For example, various types of information such as "XX table," "XX list," "XX queue," and "XX file" may be referred to as "XX information." When describing identification information, terms such as "identification information," "identifier," "name," "ID," and "number" are used, and these terms are interchangeable.

[0012] When there are multiple components with the same or similar function, they may be described using the same symbol but with different subscripts. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description.

[0013] In embodiments, processing performed by executing a program may be described. Here, the computer executes the program using a processor (e.g., CPU, GPU) and performs processing defined by the program using memory resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the main entity performing the processing by executing the program may be the processor. Similarly, the main entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The main entity performing the processing by executing the program may be an arithmetic unit, and may include a dedicated circuit that performs a specific processing. Here, a dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

[0014] The program may be installed on the computer from the program source. The program source may be, for example, a program distribution server or a storage medium readable by the computer. If the program source is a program distribution server, the program distribution server includes a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to other computers. In addition, in some embodiments, two or more programs may be implemented as a single program, or one program may be implemented as two or more programs.

[0015] In this embodiment, a technique is described that enables a user to suitably view a three-dimensional virtual image object. For example, it enables a user of an information processing device to view the object in a manner that suits the convenience and circumstances of themselves or other users of other devices.

[0016] As shown in Figure 1, the information processing device 101 is a wearable device that is worn on the user's head. The information processing device 101 is also a device that can superimpose images related to augmented reality onto real space. Examples of the information processing device 101 include AR (Augmented Reality) glasses and HMD (Head-Mounted Display). The information processing device 101 can display images using optical see-through methods, video see-through methods, etc.

[0017] Furthermore, the information processing device 101 may be, for example, a device in which an information terminal such as a smartphone or tablet is attached to a frame worn on the user's head. In this case, the information processing device 101 displays information on the display of the information terminal. In addition, some of the processing performed by the information processing device 101, which will be described later, may be executed by the processor of the information terminal.

[0018] The information processing device 101 can superimpose and display three-dimensional virtual image objects in real space. The information processing device 101 displays the three-dimensional virtual image objects so that they are placed in a specific location, such as on a table. Note that the term "virtual image object" refers to the image of the virtual object.

[0019] As shown in Figure 2, the information processing devices 101A and 101B superimpose a virtual video object OJ, such as a board game in which users A and B interact in turns, onto the real space. In this example, the upper part of the entire virtual video object OJ includes an automatically moving virtual video object OJ1 (a virtual video object of scenery such as stars and the moon). On the other hand, the lower part of the virtual video object includes a virtual video object OJ2, which can be moved by user operation. This virtual video object OJ2 includes, for example, a virtual video object of a game piece that users A and B can operate. Although there are two users in the example in Figure 2, the technology of this embodiment may also be applied when three or more users each wear an information processing device and view the virtual video object.

[0020] When users A and B are in the same real space, such as being in the same room, the information processing device 101A can display user B using an optical see-through method or a video see-through method, and therefore does not need to display a video virtual object representing user B, such as an avatar. However, even when users A and B are in the same real space, the information processing device 101A may display a video virtual object of user B superimposed on the real user B. On the other hand, when users A and B are in different real spaces, such as being in their respective homes, the information processing devices 101A and 101B superimpose and display a video virtual object OJ in the same situation at a specific location in their respective real spaces. In this case, the information processing device 101A superimposes and displays a video virtual object representing user B along with the video virtual object OJ. The information processing device 101A may obtain information about user B's video virtual object from the information processing device 101B or from an external server. Furthermore, the information processing device 101A may display user A as a video virtual object, not just user B.

[0021] An example of the configuration of the information processing device will be described with reference to Figure 3. As shown in Figure 3, the information processing device 101 includes an out-camera 111, an in-camera 112, a distance measuring sensor 113, an acceleration sensor 114, a gyro sensor 115, a geomagnetic sensor 116, a positioning sensor 117, a display 118, a microphone 119, a speaker 120, a vibrator 121, a timer 122, and an operation I / F 123.

[0022] The rear camera 111 acquires an image by converting light incident from the lens into an electrical signal using an image sensor such as a CCD image sensor or a CMOS image sensor. The rear camera 111 can be positioned, for example, to capture images of the area around the information processing device 101.

[0023] The in-camera 112 detects the user's gaze direction, for example, when the information processing device 101 is attached. The in-camera 112 may also include, for example, a left-eye gaze sensor for the left eye and a right-eye gaze sensor for the right eye. The left-eye gaze sensor detects the user's left eye and can capture the viewpoint of the left eye by detecting the movement of the left eye and the direction of the gaze. The right-eye gaze sensor detects the user's right eye and can capture the viewpoint of the right eye by detecting the movement of the right eye and the direction of the gaze.

[0024] The information processing device 101 may perform predetermined processing using gaze information acquired from the in-camera 112. The information processing device 101 may, for example, display an input operation screen such as an operation menu on the display 118. The information processing device 101 may then acquire the positions of icons, buttons, etc. on the input operation screen that the user's gaze is directed towards as input operation information and perform predetermined processing.

[0025] Furthermore, the process for detecting eye movement can utilize well-known techniques commonly used as eye-tracking processing. For example, in a method using corneal reflection, infrared light is emitted from an infrared LED (Light Emitting Diode) towards the face, and an infrared camera, which is an example of a gaze sensor, takes an image. Then, using the point of light reflection on the cornea as a reference point, eye movement (direction of the eye) can be estimated from changes in eye position and other geometric characteristics.

[0026] The distance measuring sensor 113 is a sensor that measures the distance, direction, and position of an object from the information processing device 101, and can capture the shape of an object as a three-dimensional object. Examples of distance measuring sensors 113 include LiDAR (Light Detection and Ranging), which irradiates an object with laser light such as infrared light and measures the reflected scattered light; TOF (Time Of Flight), which measures the reflection time of pulsed light irradiated onto a subject for each pixel; and millimeter-wave radar, which emits millimeter-wave radio waves and captures the reflected waves. Furthermore, the distance measuring sensor 113 may be a sensor that measures based on the angle at which reflected light from the object is received. In other words, the distance measuring sensor 113 may be a triangulation-type sensor. Also, the distance measuring sensor 113 may be a stereo camera that measures based on a parallax image, and the out-camera 111 may be a stereo camera. Furthermore, if information regarding the distance to an object, direction, and position of the object is to be obtained without using the distance measuring sensor 113, the distance measuring sensor 113 may be omitted.

[0027] The acceleration sensor 114 can detect, for example, the tilt and direction of the information processing device 101. The gyro sensor 115 is a sensor that detects angular velocity or angular acceleration in the rotational direction and can capture the state of an object's orientation. Therefore, the orientation of the information processing device 101, such as its tilt and direction, can be detected using the acceleration sensor 114 and the gyro sensor 115. The geomagnetic sensor 116 is a sensor that detects the Earth's magnetic field and can detect the direction the information processing device 101 is facing. It is also possible to detect the movement of the information processing device 101 by using a three-axis type sensor that detects the geomagnetic field in the up, down, and forward, backward, left, and right directions, thereby capturing the changes in the geomagnetic field in response to the movement of the information processing device 101. The information processing device 101 can use these sensors to acquire information about the user's orientation and movement.

[0028] The positioning sensor 117 is, for example, a GPS (Global Positioning System) receiver. In this case, by receiving satellite radio waves from GPS satellites and performing positioning calculations, it is possible to detect, for example, the coordinates on Earth where the information processing device 101 is located. Positioning calculations may be performed by the GPS receiver, but other components (such as the processor of the information processing device or an external device) may also perform positioning calculations. The information processing device 101 may also acquire location information by other methods. For example, the information processing device 101 may determine its current position by analyzing images taken by the rear camera 111 and / or three-dimensional point cloud data measured by a distance measuring sensor, and technologies such as SLAM (Simultaneous Localization and Mapping) are known. When acquiring location information without using the positioning sensor 117, the positioning sensor 117 may be omitted.

[0029] Furthermore, technologies such as VPS (Visual Positioning System) are known for determining location and other information by analyzing camera images. The information processing device 101 may acquire location information using VPS with the captured image acquired by the rear camera 111. Alternatively, the information processing device 101 may transmit the captured image acquired by the rear camera 111 to an external device. The external device may then acquire location information based on VPS using this captured image and distribute the acquired location information to the information processing device 101. Note that if location information is acquired without using the positioning sensor 117, the positioning sensor 117 may be omitted.

[0030] Furthermore, the information processing device 101 may acquire location information by PDR (Pedestrian Dead Reckoning) using, for example, an acceleration sensor 114, a gyro sensor 115, etc. If location information is acquired without using the positioning sensor 117, the positioning sensor 117 may be omitted.

[0031] The display 118 can be configured using a display such as a liquid crystal display, a touch panel display, or an organic EL display, and displays virtual video objects, camera images, etc. For example, if the information processing device 101 is an immersive HMD, the display 118 is configured with a liquid crystal panel, a display that shows holographic images, etc., and displays virtual video objects, camera images, etc.

[0032] The microphone 119 collects the user's own voice, external sounds, etc., and converts them into audio data. For example, the user may speak a voice related to an input operation or other instruction, and the information processing device 101 may acquire the audio data collected by the microphone as instruction information for the input operation or other instruction, and perform predetermined processing.

[0033] The speaker 120 can output sound based on audio data and inform the user of notification information by voice. The information processing device 101 can, for example, emit a warning message by voice from the speaker to inform the user. By providing voice notifications, usability can be improved. In addition, the speaker 120 can output sound from, for example, a video virtual object (i.e., the playback sound of the content).

[0034] The vibrator 121 generates vibrations under control from the processor and converts the notification instruction information sent to the user by the information processing device 101 into vibrations. The vibrator 121 transmits vibrations to the user's head where the information processing device 101 is worn, making the notification instruction information known to the user and improving usability.

[0035] The information processing device 101 may use a timer 122 to manage processing time, processing start time, etc.

[0036] The operation I / F 123 (operation interface) is an interface that realizes input means such as gaze, hand, pointer, and also keyboard, key buttons, touch keys, etc., which are operating devices separate from the information processing device 101, and information that the user wants to input is input via the operation I / F 123.

[0037] The information processing device 101 may display an input operation screen, such as an operation menu, on the display screen of the display 118, capture the position on the input operation screen where the user's gaze is directed as input operation information, and perform an action according to the input operation information. Alternatively, the information processing device 101 may display a cursor, such as a hand or pointer, on the input operation screen, capture the user's cursor operations as input operation information, and perform an action according to the input operation information. Furthermore, the information processing device 101 may capture the user's gestures as input operation information and perform an action according to the input operation information. In the case of input using an input device such as a keyboard, key buttons, or touch keys, the input device may be connected to the main unit of the information processing device 101 using a wired cable or wirelessly.

[0038] Examples of user input include inputting various types of information. For example, user-specified information, as described later, is entered by the user. Other examples of user input include applications and various settings operations such as volume adjustment.

[0039] Furthermore, the information processing device 101 includes a processor 131, a memory 132, and a communication device 133. The processor 131 is the main component that executes predetermined processes and is configured using an appropriate semiconductor device such as a CPU (Central Processing Unit). As shown in the figure, in the information processing device 101, each component is connected to the processor 131 via a bus 141. The processor 131 controls each component of the information processing device 101 by executing the operating system (OS) and application programs for operation control stored in the memory 132, thereby realizing the functions of the OS, middleware, and applications.

[0040] Memory 132 consists of main memory and non-volatile memory, and stores various programs 151 and information data (data 152) handled by the processor 131, etc. An example of information data will be described later. Memory 132 stores programs 151 such as the operating system and various operation control application programs used by the processor 131. Memory 132 may also store various settings and user-defined settings.

[0041] The communication device 133 is a device relating to the interface used by the information processing device 101 for communication with the outside world. The communication device 133 may include, for example, a short-range wireless communication interface, a network communication interface, and the like.

[0042] A short-range wireless communication interface is a communication interface for performing short-range wireless communication with devices that are within range of short-range wireless communication. Short-range wireless communication is performed, for example, using electronic tags, but is not limited to this. If the information processing device 101 is near a device and this device is at least capable of wireless communication, short-range wireless communication may be performed using Bluetooth®, IrDA (Infrared Data Association®), Zigbee®, HomeRF (Home Radio Frequency®), or wireless LAN (IEEE 802.11a, IEEE 802.11b, IEEE 802.11g).

[0043] The network communication interface is a communication interface that performs wireless communication with external information servers and other devices via an external network, such as through base station communication. The network communication interface can receive various types of information stored in the information server. Long-range wireless communication such as W-CDMA (Wideband Code Division Multiple Access) and GSM (Registered Trademark) (Global System for Mobile Communications) can be used as base station communication.

[0044] Further, the information processing apparatus 101 may include a wired connection input / output interface. The wired connection input / output interface is configured for wired connection and is connected to a device at a wired connection destination via a wired cable. Also, the information processing apparatus 101 can include a battery that serves as a power source. Then, a charging device or the like is connected as a device at the wired connection destination, and the wired connection input / output interface may be an input / output interface that inputs or outputs power via a wired cable. Also, a battery may be connected to the information processing apparatus 101 as a device at the wired connection destination, and the wired connection input / output interface may be an input / output interface that inputs or outputs power via a wired cable. Also, the information processing apparatus 101 may be connected to an external power source and power may be supplied from the wired connection input / output interface. Also, the wired connection input / output interface may be used for communication with a device at the wired connection destination. Also, the wired connection input / output interface may be configured to be connected by Power over Ethernet (PoE; Ethernet is a registered trademark).

[0045] Note that devices that can be wired-connected using the wired connection input / output interface include, in addition to the charging device, computer devices such as personal computers, audio devices such as earphones or loudspeakers, and input operation devices such as keyboards, key buttons, and touch keys.

[0046] An example of information data and an example of the functional configuration of the information processing apparatus will be described while referring to FIG. 4.

[0047] As shown in FIG. 4, as an example, the memory 132 can store virtual object information 401, position information 402, audio information 403, time information 404, user operation information 405, gaze information 406, audiovisual assistance control information 407, and other information 408 as information data. Note that these pieces of information are not limited to being stored in the memory 132 and may be stored in an external server.

[0048] Also, as an example, the information processing apparatus 101 includes a virtual object processing unit 411, a position information processing unit 412, an audio information processing unit 413, a time information processing unit 414, a user motion information processing unit 415, a gaze information processing unit 416, a viewing assistance control processing unit 417, and other information processing unit 418. These configurations are realized by the processor 131 storing data such as programs in the main storage device of the memory 132 and performing data processing. Also, processing units such as the virtual object processing unit 411 acquire information such as virtual object information 401 from the memory 132. Note that these processing units may acquire information such as virtual object information 401 from other information processing apparatuses or external servers via the communicator 133, or share this information with other information processing apparatuses or external servers. Also, in realizing these configurations, other hardware configurations (such as hardware configurations of sensors, interface devices, timers, etc.) may be used.

[0049] The virtual object processing unit 411 acquires virtual object information 401. The virtual object information 401 relates to information on the virtual object to be displayed and includes information on the video virtual object that the information processing apparatus 101 superimposes and displays in the real space. Also, the virtual object processing unit 411 uses the virtual object information 401 to perform processing related to the display of the video virtual object.

[0050] The location information processing unit 412 acquires location information 402. Location information 402 is information regarding the position of the information processing device 101 detected by the positioning sensor 117 and the position of the virtual video object. Furthermore, location information 402 is not limited to information in world coordinates indicating where on Earth it is located, but may also be information indicating a relative position based on the position of the information processing device 101 or the position of the virtual video object OJ. In addition to information regarding position, information regarding direction may also be stored as location information 402. For example, information regarding the direction of the user (person) in real space relative to the virtual video object OJ, or information regarding the direction of the virtual video object representing the user may be stored. It may also include information regarding the position and direction of other information processing devices received by the communication device 133. The location information processing unit 412 may also acquire the position and direction of the virtual video object representing the user as location information 402 from the application used to experience the virtual video object OJ.

[0051] The audio information processing unit 413 acquires audio information 403. The audio information 403 is information related to audio, and may include, for example, information about the audio of a video virtual object (i.e., the audio of the content being played). In addition, for example, when experiencing a video virtual object, the microphone 119 may collect the user's voice and include the information of the collected audio in the audio information 403. It may also include audio of content being played on another information processing device or information indicating the user's voice, which has been received by the communication device 133.

[0052] The time information processing unit 414 acquires time information 404. The time information 404 is information about the time during the experience of the video virtual object, and may include, for example, time information (playback time) indicating the playback position of the video virtual object OJ. The time information 404 may also include information indicating the time when playback of the video virtual object OJ was stopped, as measured using the timer 122, and the duration of the stop. The time information 404 may also include information about the playback time and stop time of the video virtual object OJ received by the communication device 133 from other information processing devices.

[0053] The user action information processing unit 415 acquires user action information 405. User action information 405 is information about the user's actions when experiencing a video virtual object. User action information 405 may include, for example, information indicating the user's operation input to the video virtual object (for example, a piece operated by the user). It may also include information indicating the user's actions on other information processing devices, received by the communication device 133. If the user is a person in real space, the user action information 405 relates to the actions of the user (person) in real space. On the other hand, if the user is a video virtual object, the user action information 405 relates to the actions of the video virtual object representing the user. The user action information processing unit 415 may also acquire user action information 405 from the application used to experience the video virtual object OJ and store it in memory 132. The user action information processing unit 415 may also acquire user action information 405 of users on other information processing devices via the communication device 133.

[0054] The gaze information processing unit 416 acquires gaze information 406. The gaze information 406 is information about the user's gaze during the experience of the virtual video object. The gaze information 406 is, for example, information about the user's gaze direction detected by the in-camera 112. It may also include information indicating the gaze of a user of another information processing device, received by the communicator 133. This information relates to the gaze of a user (person) in real space toward the virtual video object, or to the gaze of the virtual video object representing the user. If the user is a person in real space, the gaze information 406 relates to the gaze of the user (person) in real space. On the other hand, if the user is a virtual video object, the gaze information 406 relates to the gaze of the virtual video object representing the user. The gaze information processing unit 416 may also acquire gaze information 406 from an application used to experience the virtual video object OJ. The gaze information processing unit 416 may also acquire the gaze information 406 of a user of another information processing device via the communicator 133.

[0055] The viewing assistance control processing unit 417 acquires viewing assistance control information 407. The viewing assistance control information 407 is information regarding the target of viewing assistance control, such as stopping and playing, as described later, or the content of the viewing assistance control. The viewing assistance control information 407 may be information set by the user using the operation I / F 123, etc., or it may be information pre-registered in memory 132 or an external server. The viewing assistance control processing unit 417 may also acquire the viewing assistance control information 407 from the application used to experience the video virtual object OJ.

[0056] The other information processing unit 418 acquires other information 408. Other information 408 is other information used to control the information processing device 101. Other information 408 includes, for example, image information, distance measurement information, communication information, and operation input information. As an example of processing, the other information processing unit 418 acquires image information showing the real space captured by the out camera 111, and uses the acquired image information to perform processing related to displaying the image showing the real space.

[0057] An example of viewing assistance control information will be explained with reference to Figure 5. Viewing assistance control information 407 may, for example, be information that targets the entire video virtual object OJ as the subject of viewing assistance control (setting 1). Viewing assistance control information 407 may, for example, be information that targets user-operable video virtual objects in the video virtual object OJ (for example, video virtual objects that the user can individually move) as the subject of viewing assistance control (setting 2). Viewing assistance control information 407 may, for example, be information that targets non-scenery video virtual objects in the video virtual object OJ as the subject of viewing assistance control (setting 3). Viewing assistance control information 407 may, for example, be information that targets video virtual objects in a predetermined area of ​​the video virtual object OJ (for example, lower video virtual objects including video virtual objects of pieces or characters operated by the user) as the subject of viewing assistance control (setting 4). Viewing assistance control information 407 may, for example, be information that targets audio in the video virtual object OJ as the subject of viewing assistance control (setting 5). The viewing assistance control information 407 may, for example, be information that targets virtual video objects related to user actions for viewing assistance control (setting 6). The viewing assistance control information 407 may, for example, be information that targets user voice for viewing assistance control (setting 7). The viewing assistance control information 407 may, for example, be information that targets objects related to predetermined supplementary information for viewing assistance control (setting 8). The viewing assistance control information 407 may, for example, be information that targets virtual video objects OJ displayed by a specific information processing device 101 among the information processing devices 101 worn by each user experiencing the virtual video objects for viewing assistance control (setting 9). The viewing assistance control information 407 may, for example, be information that targets virtual video objects OJ shared by multiple information processing devices 101 for viewing assistance control (setting 10).

[0058] Referring to Figure 6, an example of the basic processing flow of the information processing device will be explained. In detail, the main processing unit is the processor 131.

[0059] The processor 131 of the information processing device 101A displays the video virtual object OJ in a temporary position, and then places the video virtual object OJ at a specific location specified by user A using the operation I / F 123 (S601). Then, when user A inputs a playback command using the operation I / F 123, the processor 131 plays the video virtual object OJ, which is the content (S602). In addition to the specification by user A via the operation I / F 123, the information processing device 101B or other device may also receive information from another device via the communication device 133 indicating the location of the video virtual object OJ or a playback command for the video virtual object OJ, and the processor 131 may place or play the video virtual object OJ based on the received information. In this case, it is desirable that the processor 131 notifies user A about the placement or playback via the display 118 or speaker 120, and then places or plays the video virtual object OJ after user A confirms it via the operation I / F 123.

[0060] The processor 131 then determines whether the position of the information processing device 101A or the information processing device 101B has moved outside the range PA, which is a predetermined distance from a specific location where the virtual video object OJ is placed, due to the movement of the user wearing the information processing device 101 (S603). For example, the processor 131 determines whether the information processing device 101A is located outside or inside the range PA based on position information acquired using the positioning sensor 117 or the acceleration sensor 114 or gyro sensor 115. The processor 131 also receives position information of the information processing device 101B transmitted from the information processing device 101B via the communicator 133 at predetermined intervals and determines whether the information processing device 101B is located outside or inside the range PA. The communicator 133 may also receive determination information from the information processing device 101B indicating that it has moved outside or inside the range PA. Furthermore, if user B, who is wearing the information processing device 101B, is in the same real space, the processor 131 may use the image captured by the out-camera 111 to determine whether the information processing device 101B has moved outside the range PA.

[0061] If the information processing device 101A or the information processing device 101B determines that it has moved outside the range PA (S603-YES), the processor 131 performs viewing assistance control 1 on the viewing assistance control target based on the setting of the viewing assistance control information 407 (S604). Viewing assistance control 1 is a control related to stopping a video virtual object OJ, or recording time such as the stop time. For example, if the setting of the viewing assistance control information 407 corresponding to the video virtual object OJ being played is setting 2 in Figure 5, the processor 131 performs viewing assistance control 1 such as stopping the playback of the user-operable video virtual object OJ2 among the video virtual objects OJ in Figure 2. On the other hand, for the automatically moving video virtual object OJ1, which is not included as a control target in setting 2, the processor 131 does not perform viewing assistance control 1 and continues playback.

[0062] Depending on the type of video virtual object content, stopping the entire playback may cause a sudden change in the user's immersion in the video virtual object when they resume viewing it. In such cases, it may be preferable to stop only a portion of the playback. To adjust the degree of change in immersion, it is preferable to set multiple variations of the portion to be controlled by viewing assistance. On the other hand, depending on the type of video virtual object, stopping the entire playback may be more appropriate than stopping a portion, as it allows the user to understand the situation more easily. As in this embodiment, by classifying video virtual objects, setting viewing assistance control targets, and performing viewing assistance control on the set targets, users can view the video virtual object more favorably.

[0063] In S604, it is desirable for the processor 131 to transmit both or either of the following to the information processing device 101B via the communicator 133: information indicating that the information processing device 101A or the information processing device 101B has determined to have moved outside the range PA, and information indicating that viewing assistance control 1 has been performed on the viewing assistance control target. For example, when the information processing device 101A transmits information indicating that it has determined that the information processing device 101A has moved outside the range PA, or information indicating that viewing assistance control 1, such as stopping the playback of the video virtual object OJ2, has been performed, it is desirable for the information processing device 101B to display a message such as "User A: Stopped" to notify that User A's operation is stopped or that User A is away from their seat. This prevents misunderstandings between users. Alternatively, for example, each device may use the position information of the information processing device 101A and the information processing device 101B to determine whether the other device has moved outside the range PA and perform the corresponding control. In this case, the processor 131 can omit sending information to the information processing device 101B.

[0064] Then, the processor 131 determines whether the information processing device 101A or the information processing device 101B, which was outside the range PA, has moved inside the range PA (S605). The determination in S605 is performed in the same way as in S603.

[0065] If the processor 131 determines that the information processing device 101 outside the range PA has moved inside the range PA (S605-YES), it performs viewing assistance control 2 on the target for which viewing assistance control 1 was performed in S604 (S606). Viewing assistance control 2 is a control such as resuming playback of the video virtual object OJ or rewinding playback from a recorded stop time. For example, if the processor 131 stops playback of the video virtual object OJ2 that is moved by user operation in viewing assistance control 1, it releases the stop of operation by viewing assistance control 1 and resumes playback of the video virtual object OJ2 in viewing assistance control 2. Alternatively, the processor 131 records the playback time of the video virtual object OJ in viewing assistance control 1, and performs rewinding playback of the three-dimensional video virtual object OJ in viewing assistance control 2 based on the recorded playback time.

[0066] Furthermore, when resuming or rewinding playback of the video virtual object OJ, the processor 131 of the information processing device 101A may add video virtual objects that indicate the movements of the other user (user B of the information processing device 101B) based on user operation information 405, or sound based on sound information 403. The processor 131 may also increase the playback speed when resuming or rewinding playback. The processor 131 may also notify the user that rewinding playback is in progress through sound, vibration, etc. The processor 131 may also change the display format of the video virtual object that is the target of viewing assistance control (for example, the video virtual object OJ2 that has been operated by the user) such as size, color, and decoration during playback. This can guide the user to focus on the playback portion and assist the user in confirming the playback portion.

[0067] Furthermore, when resuming playback or rewinding playback of the video virtual object OJ, the processor 131 may increase the playback speed. For example, if the processor 131 determines that the rewind playback time is longer than a predetermined time, it may increase the playback speed compared to the normal speed. In this case, the processor 131 may provide a notification indicating that playback is being increased, or a notification indicating that playback is currently being increased. For example, it may provide a message such as "Playing at double speed" via voice, display, etc.

[0068] For example, if only the information processing device 101A, which has returned to the range PA, performs playback and rewind playback of the video virtual object OJ at an accelerated playback speed, the processor 131 acquires information regarding the playback time of the video virtual object OJ in the information processing device 101B. Then, if the playback time of the video virtual object OJ in the information processing device 101A reaches the playback time of the video virtual object in the information processing device 101B, the processor 131 may slow down the playback speed of the video virtual object OJ and change it back to the normal speed. In addition, even if three or more users are each wearing an information processing device and viewing the video virtual object, only one information processing device may perform playback and rewind playback of the video virtual object OJ at an accelerated playback speed. In this case, the processor 131 of the information processing device that has returned to the range PA acquires information regarding the playback time of the video virtual object OJ in the other information processing devices. Then, if the playback time of the video virtual object OJ reaches the playback time that is most advanced among the other information processing devices, the processor 131 may slow down the playback speed of the video virtual object OJ and change it back to the normal speed.

[0069] Furthermore, depending on the content of the portion of the video virtual object OJ from which playback is resumed or rewound, the playback speed of the resumed or rewound playback may be changed, or some time periods may be skipped during playback. In addition, the playback time and content of the 3D video virtual object OJ may be recorded periodically, and during periods when no operation is performed, the playback speed may be increased compared to periods when an operation is performed, or periods when no operation is performed may be skipped during playback.

[0070] Alternatively, when the processor 131 executes the viewing assistance control 1, it may determine whether the target portion of the viewing assistance control has been modified by a user of another information processing device, and record the time the modification was made. Then, in the viewing assistance control 2, playback speed may be increased up to the time before the modification was made, or playback may be resumed from the time the modification was made.

[0071] In S606, it is desirable for the processor 131 to transmit to the information processing device 101B via the communicator 133 both or either of the following: information indicating that the information processing device 101A or the information processing device 101B has determined to have moved inside the range PA, and information indicating that viewing assistance control 2 has been performed on the viewing assistance control target. For example, it is desirable for the information processing device 101A to transmit to the information processing device 101B information indicating that it is in the process of resuming playback of the viewing assistance control target, and for the information processing device 101B to display a message 701 such as "User A: Checking" or "User A: Rewinding Playback in Progress" as shown in Figure 7, so that user B can recognize the status of other users. Alternatively, the information processing device 101A may transmit to the information processing device 101B information regarding the video virtual object OJ that is being rewound and played back in the information processing device 101A, and the information processing device 101B may display a copy 702 indicating the video virtual object that is being rewound and played back in the information processing device 101A.

[0072] Alternatively, for example, the position information of the information processing devices 101A and 101B may be used to determine whether other devices have moved inside the range PA, and the corresponding control may be performed by each device. In this case, the processor 131 can omit transmitting information to the information processing device 101B.

[0073] Alternatively, the information processing device 101A or 101B may transmit display data for the virtual video object OJ being played back to the display device or an external server, and the display device may display the display data received from the information processing device 101 or the external server. Alternatively, as shown in Figure 8, the camera 802 may capture images of the area around the information processing device 101, transmit the captured images to the display device 801 or an external server, and display the display data along with the captured images on the display device 801. Examples of the display device 801 include information processing devices 801A such as AR glasses or HMDs, televisions 801B, personal computers, and smartphones 801C. By displaying the display data from the information processing device 101 on the display device in this way, the situation can be shared with people in other locations, such as spectators of a game.

[0074] Here, with reference to Figure 9, an example of viewing assistance control will be specifically explained. Figure 9 shows a situation where User A, wearing the information processing device 101A, performs an operation on the virtual video object OJ2 (in this example, a piece that is an operable object), and then moves outside the range PA. In other words, it shows a situation where the information processing device 101A moves outside the range PA after User A has finished their turn.

[0075] In this example, the setting of the viewing assistance control information 407 corresponding to the video virtual object OJ is setting 2 in Figure 5, and when the information processing device 101A moves outside the range PA, the processor 131 stops playback of the video virtual object OJ2 as viewing assistance control 1. In this case, even if a user operation is performed on the video virtual object OJ2 via the operation I / F 123, the processor 131 controls it so that the operation is not accepted. The processor 131 may also notify that viewing assistance control has been performed by using the speaker 120, vibrator 121, and display 118, through sound, vibration, message display, etc. Furthermore, it is desirable that the information processing device 101B also controls it so that it does not accept user operations on the video virtual object OJ2, which is the target of viewing assistance control, based on the information transmitted from the information processing device 101A indicating that viewing assistance control 1 has been performed on the target of viewing assistance control.

[0076] The viewing assistance control described above prevents user A from accidentally operating the information processing device 101A even after it has moved outside the range PA. Furthermore, by transmitting information to the information processing device 101B indicating that viewing assistance control 1 has been performed on the target of viewing assistance control, it becomes possible to control the information processing device 101B so that it does not accept user operations. This prevents user A from missing user B's operation on the information processing device 101B while the information processing device 101A is returning inside the range PA.

[0077] Furthermore, it is not limited to the case where the information processing devices 101A and 101B cooperate and synchronize the control of the video virtual object OJ displayed by each device; each device may also perform viewing assistance control independently. In the case of independent control, for example, when the information processing device 101A moves outside the range PA, the information processing device 101B does not perform control to stop accepting user operations, and user B can continue operations. Even in such a case, the processor 131 in the information processing device 101A records the stop time as viewing assistance control 1, and performs rewind playback from the recorded stop time as viewing assistance control 2, thereby preventing user A from missing user operations on the information processing device 101B.

[0078] Furthermore, by partially stopping playback of the video virtual object OJ rather than stopping the entire playback (for example, by not applying viewing assistance control to a video virtual object OJ1 that automatically moves, such as a landscape of stars or the moon), it is possible to suppress the decrease in the sense of presence that users within the PA range feel towards the video virtual object OJ. In addition, users who have moved outside the PA range can recognize from a distance that the content of the video virtual object OJ itself is continuing. Moreover, it is possible to suppress abrupt changes in the sense of presence towards the video virtual object OJ while assisting users who have moved outside the PA range to naturally return to viewing the video virtual object OJ.

[0079] Furthermore, if the video virtual object OJ is content such as a board game in which users interact in turns, the information processing device 101 of the user who performed the operation (i.e., the user who finished their turn) may display a message M1 such as, "You have performed an operation." In this case, the video virtual object OJ2 operated by the user will be subject to viewing assistance control 1 and viewing assistance control 2, while the playback of the object in message M1 will not be stopped. Message M1 may be displayed for a predetermined time and then hidden.

[0080] Furthermore, if the video virtual object OJ is turn-based content, and user B cannot perform any additional operations after operating once until user A also operates, the control may be modified depending on the type of video virtual object OJ, such as not restricting user operations.

[0081] Even if the information processing device 101 is determined to have moved from outside the range PA to inside in S605, the user may not be able to see the part of the video virtual object that is the target of viewing assistance control (for example, the three-dimensional video virtual object OJ2 on which the operation was performed). For this reason, the user who has returned to the inside of the range PA may be notified of at least one of the following: that a change has been made, the location of the change, and the location and direction in which the change can be seen. As shown in Figure 10, this notification is performed, for example, by displaying a video virtual object such as an arrow 1001 or symbol 1002 that indicates the location and direction of the change, displaying a message 1003, or displaying a video virtual object 1004 that indicates the location in which the change can be seen. In addition, although not shown, a video virtual object such as a frame containing the change may also be displayed. This guides the user who has moved inside the range PA to focus on the part that is the target of viewing assistance control, and helps the user to confirm the part of the video virtual object OJ on which the operation was performed.

[0082] When such a notification is made, for example as shown in Figure 11, the processing is performed by adding S1107 and S1108 to the processing flow in Figure 6. If the processor 131 determines that the information processing device 101 has moved inside the range PA (S605-YES), it determines whether the target part for which viewing assistance control 1 was performed in S604 is visible to the user (S1107). The processor 131 determines whether the user is able to see the target part based on the user's gaze direction detected by, for example, the in-camera 112. If the processor 131 determines that the target part is not visible to the user (S1107-NO), it sends a notification regarding the change to the target part, including at least one of the fact that a change has been made, the location of the change, and the position or direction in which the change can be seen (S1108). Then, if the processor 131 determines that the target part is visible to the user (S1107-YES), it performs viewing assistance control 2 on the target for which viewing assistance control 1 was performed in S604 (S606). Furthermore, not limited to the processing example in Figure 11, if the processor 131 determines that the target portion is not visible to the user (S1107-NO), it may, instead of notifying the user of the change, perform rewind playback on the target portion from the recorded playback time. Alternatively, it may perform rewind playback in addition to notifying the user of the change. The processor 131 may also change the display format of the video virtual object, such as its size, color, and decorations, which is the target of the viewing assistance control, before performing rewind playback.

[0083] Furthermore, for users who have returned to the area PA, a gaze range may be displayed to indicate the area (region) to focus on, such as the video virtual object that is the target of viewing assistance control (for example, the video virtual object that has been operated on). For example, the gaze range can be indicated to the user by displaying video virtual objects such as arrows, symbols, or frames that indicate the position or range of the part to which viewing assistance control has been performed. Alternatively, the gaze range can be indicated to the user by changing the display format of the video virtual object that is the target of viewing assistance control, such as its size, color, or decoration. This helps to guide users who have moved into the area PA to focus on the part to which viewing assistance control is performed, and to help the user confirm the part of the video virtual object OJ that has been operated on.

[0084] When displaying such a gaze range, for example, as shown in Figure 12, the processing is performed by adding S1207 and S1208 to the processing flow in Figure 6. When the processor 131 determines that the information processing device 101 has moved inside the range PA (S605-YES), it displays a virtual video object indicating the gaze range at the position corresponding to the target part for which viewing assistance control 1 was performed (S1207). The processor 131 then determines whether the user has viewed the target part for viewing assistance control 1 (S1208). The processor 131 determines whether the user has viewed the target part based on the direction of the user's gaze detected by, for example, the in-camera 112. When the processor 131 determines that the user has viewed the target part for viewing assistance control 1 (S1208-YES), it performs viewing assistance control 2, such as resuming playback of the target part for which viewing assistance control 1 was performed (S606).

[0085] Furthermore, in the processing flow of Figure 6, if the processor 131 determines that the information processing device 101 has moved inside the range PA (S605-YES), it may determine whether there have been any changes, such as operations, by users of other information processing devices on the video virtual object that is the target of viewing assistance control (for example, the video virtual object that has been operated on). If it determines that there has been a change, viewing assistance control 2 is performed in S606, and information indicating changes, such as operations, by users of other information processing devices may be displayed, as shown in Figure 13. In the example of Figure 13, messages (1301, 1302) indicating information on operations by users B and C are added and played back. This guides the user who has moved inside the PA (user A in this example) to confirm the part of the video virtual object OJ that has been operated on by other users, and quickly reduces the difference in recognition between users regarding the video virtual object OJ.

[0086] Referring to Figure 14, an example of viewing assistance control will be specifically explained. Figure 14 shows a situation where a user moves outside a predetermined distance range, and another user manipulates a three-dimensional virtual image object.

[0087] In this example, when the information processing device 101A moves outside the PA range, it displays and plays a copy 1401 of the video virtual object OJ in the vicinity of the information processing device 101A after it has moved. When the information processing device 101A returns to the PA range, it stops displaying the copy 1401 of the video virtual object OJ. This allows the user to see the process (such as changes) of parts of the video virtual object OJ that have been manipulated by other users, even when the user has moved outside the PA range.

[0088] Furthermore, the information processing device 101A may display the copy 1401 of the video virtual object OJ within a predetermined distance range from the information processing device 101A, regardless of whether it has moved outside the range PA. In addition, the information processing device 101A may allow the copy 1401 of the video virtual object to be operated separately from the original video virtual object OJ. This allows, for example, user A of the information processing device 101A to separately check the portion of the video virtual object OJ that has been operated by another user using the copy 1401, and to freely examine the process.

[0089] Although embodiments have been described above, the present invention is not limited to the embodiments described above, and various modifications and equivalent configurations are included. For example, in S603 of Figure 6, it is not limited to whether the user of the information processing device 101 has moved outside the range PA, but it is possible to determine whether the user has performed a first action. The first action may include, for example, whether the user has removed the information processing device 101 from their head, or whether the user has performed a predetermined operation on the video virtual object OJ, such as instructing it to stop playback or to record the playback time. Also, in S605, it is not limited to whether the user of the information processing device 101 has moved inside the range PA, but it is possible to determine whether the user has performed a second action. The second action may include, for example, whether the user has put on the information processing device 101, or whether the user has performed a predetermined operation on the video virtual object OJ, such as instructing it to resume playback or to rewind playback. Furthermore, in S603, instead of determining whether the first action has been performed, the information processing device 101 may determine whether the user is no longer visually viewing the video virtual object OJ. Alternatively, instead of determining whether the second operation was performed in S605, it may be determined whether the user has entered a state where they can view the video virtual object OJ.

[0090] Furthermore, for example, the embodiments described above are explained in detail to make the present invention easier to understand, and the present invention is not necessarily limited to having all the configurations described. Also, for example, some of the configurations of the embodiments may be added, deleted, or replaced with other configurations. Also, the numbers and messages that appear in the text and figures are merely examples, and are not particularly limited as long as the effects of the present invention are not impaired by using different ones. Also, the programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. Also, the order in which each processing is performed may be changed as long as the effects of the present invention are not impaired. Furthermore, some or all of the above configurations, functions, processing units, processing means, etc. may be implemented in hardware, for example, by designing them as integrated circuits, general-purpose processors, or application-specific processors. A processor includes transistors and other circuits and is considered a circuit or processing circuit. Also, the above configurations, functions, etc. may be implemented in software by having the processor interpret and execute programs that realize each function. The information such as programs and tables that implement each function may be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD, or it may be stored in a device on a communication network. Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines in the actual product. In practice, it can be assumed that almost all components are interconnected.

[0091] 101 Information processing unit 111 Rear camera 112 Front camera 113 Distance sensor 114 Accelerometer 115 Gyroscope 116 Geomagnetic sensor 117 Positioning sensor 118 Display 119 Microphone 120 Speaker 121 Vibrator 122 Timer 123 Operation interface 131 Processor 132 Memory 133 Communication device

Claims

1. An information processing device to be attached to a user, comprising: a processor that controls the playback of a video virtual object; and a memory that stores control information indicating control over a predetermined object included in the video virtual object, wherein the processor plays the video virtual object, and when it determines that the user has performed a first action, it performs a first control over the predetermined object based on the control information stored in the memory, and continues to play the video virtual object other than the predetermined object; and when it determines that the user has performed a second action, it performs a second control over the predetermined object based on the control information stored in the memory.

2. An information processing apparatus according to claim 1, further comprising an operation interface into which instructions from the user are input, wherein the processor places and plays the virtual video object at a position instructed by the user via the operation interface, the first operation is the user moving out of a range that is a predetermined distance from the placement position of the virtual video object, and the second operation is the user moving inside the range.

3. An information processing apparatus according to claim 2, wherein the processor, when it determines that the user has moved outside the range, performs the first control on the predetermined object and displays a copy of the virtual video object near the user outside the range, and when it determines that the user has entered inside the range, performs the second control on the predetermined object and stops displaying the copy.

4. An information processing apparatus according to claim 1, wherein the first control is stopping the playback of the predetermined object or recording the playback time of the predetermined object, and the second control is restarting the playback of the predetermined object or rewinding the playback of the predetermined object.

5. An information processing apparatus according to claim 1, further comprising an operation interface into which instructions from the user are input, wherein the predetermined object is a virtual video object that can be operated by the user via the operation interface.

6. An information processing apparatus according to claim 5, wherein the processor, when it determines that the user has performed the first operation, performs the first control on the predetermined object and controls the predetermined object so as not to accept user operations on the predetermined object via the operation interface.

7. An information processing apparatus according to claim 1, further comprising a camera for photographing the user, wherein the processor determines that the user has performed the second operation and that the user can see the predetermined object using the image captured by the camera, and then performs the second control on the predetermined object based on the control information stored in the memory.

8. An information processing device to be attached to a user, comprising: a communication interface for communicating with other devices; a processor for controlling the playback of a video virtual object in cooperation with the other devices; and a memory for storing control information indicating control over a predetermined object included in the video virtual object, wherein the processor plays the video virtual object, and when it determines that the user has performed a first action, it performs a first control over the predetermined object based on the control information stored in the memory, continues playback of the video virtual objects other than the predetermined object, transmits to the other devices via the communication interface information indicating that the user has performed the first action or that the first control has been performed over the predetermined object, and when it determines that the user has performed a second action, it performs a second control over the predetermined object based on the control information stored in the memory, and transmits to the other devices via the communication interface information indicating that the user has performed a second action or that the second control has been performed over the predetermined object.

9. An information processing apparatus according to claim 8, wherein the first control is stopping the playback of the predetermined object, the second control is restarting the playback of the predetermined object, the processor, when it determines that the user has performed the first operation, stops the playback of the predetermined object, stores in the memory the time of modification made to the predetermined object by the other device, and when it determines that the user has performed the second operation, plays the predetermined object at an accelerated playback speed up to before the modification time, or restarts playback from the modification time, based on the modification time stored in the memory.

10. An information processing apparatus according to claim 8, wherein the first control is stopping the playback of the predetermined object, the second control is restarting the playback of the predetermined object, the processor, when it determines that the user has performed the first operation, stops the playback of the predetermined object and stores in the memory the modified parts of the predetermined object that have been modified by the other device, and when it determines that the user has performed the second operation, restarts the playback of the predetermined object and displays an object indicating the modified parts or an object indicating a position where the modified parts can be seen by the user, based on the modified parts stored in the memory.

11. An information processing apparatus according to claim 8, wherein the first control is stopping the playback of the predetermined object, the second control is restarting the playback of the predetermined object, the processor, when it determines that the user has performed the first operation, stops the playback of the predetermined object and stores in the memory the modified parts of the predetermined object that have been modified by the other device, and when it determines that the user has performed the second operation, restarts the playback of the predetermined object and changes the display format of the predetermined object corresponding to the modified parts stored in the memory.

12. An information processing apparatus according to claim 8, wherein the processor, upon receiving information from the other device via the communication interface indicating that the first operation has been performed by a user of the other device or that the first control has been performed on a predetermined object, provides a notification indicating that the first control is being performed on the other device; and upon receiving information from the other device via the communication interface indicating that the second operation has been performed by a user of the other device or that the second control has been performed on a predetermined object, provides a notification indicating that the second control is being performed on the other device.

13. An information processing apparatus according to claim 12, wherein the first control is stopping the playback of the predetermined object, the second control is restarting the playback of the predetermined object, and the processor displays a copy of the video virtual object that is playing the predetermined object in the other device.

14. An information processing method used in an information processing device, characterized by: storing control information indicating control over a predetermined object included in a virtual video object; playing back the virtual video object; when the user of the information processing device determines that a first action has been performed, performing a first control over the predetermined object based on the stored control information, continuing to play back the virtual video objects other than the predetermined object; and when the user determines that a second action has been performed, performing a second control over the predetermined object based on the stored control information.

15. An information processing method according to claim 14, wherein the processor places the virtual video object at a predetermined position and plays it back; the first operation is the user moving out of a range that is a predetermined distance from the predetermined position; the second operation is the user moving inside the range; the first control is stopping the playback of the predetermined object or recording the playback time of the predetermined object; and the second control is resuming the playback of the predetermined object or rewinding the playback of the predetermined object.

Images