Information processing method, information processing device, and program

Abstract

An information processing according to the present disclosure generates guide information indicating the direction of movement of a moving object according to a state of the moving object, changes the position of text information included in the guide information on the basis of a display mode of the generated guide information, and displays the guide information including the text information for which the position has been changed.

Description

Information Processing Method, Information Processing Apparatus and Program

[0001] The present disclosure relates to an information processing method, an information processing apparatus and a program.

[0002] For example, when remotely operating a vehicle, there is known a technology for assisting an operator's remote operation by superimposing and displaying guide information indicating the traveling direction of the vehicle on an image captured by a camera that images the traveling direction of the vehicle. As an example of such technology, there is also known a technology for generating guide information by reflecting information such as the steering angle of a remotely operable vehicle, a camera coordinate system, and the like.

[0003] Japanese Patent Application Laid-Open No. 2020-132431

[0004] However, in the related art, depending on the display mode of the guide information, the visibility of text information included in the guide information (for example, numbers indicating a unit amount when dividing the guide information into a plurality of parts) may be impaired. Therefore, there is a possibility that the operator's operation cannot be appropriately assisted.

[0005] An object of the present disclosure is to provide an information processing method, an information processing apparatus, and a program capable of suppressing impairment of the visibility of guide information.

[0006] In order to achieve the above object, the information processing method of the present disclosure generates guide information indicating a traveling direction of a moving object according to a state of the moving object, changes a position of text information included in the guide information based on a display mode of the generated guide information, and displays the guide information including the text information whose position has been changed.

[0007] According to the present disclosure, impairment of the visibility of guide information can be suppressed. Note that the effects described herein are not necessarily limited, and any effect described in the present specification may be obtained.

[0008] Figure 1 is a diagram illustrating an example of an information processing system including an information processing device according to an embodiment. Figure 2 is a diagram illustrating an example of the data configuration stored in the storage unit. Figure 3 is a diagram illustrating an example of a calibration method. Figure 4 is a diagram illustrating an example of guide information generated by the guide information generation unit. Figure 5 is a diagram illustrating an example of guide information generated by the guide information generation unit. Figure 6 is a diagram illustrating an example of guide information generated by the guide information generation unit. Figure 7 is a diagram illustrating an example of guide information generated by the guide information generation unit. Figure 8 is a diagram illustrating an example of guide information generated by the guide information generation unit. Figure 9 is a diagram illustrating an example of guide information generated by the guide information generation unit. Figure 10 is a diagram illustrating the monitor coordinate system. Figure 11 is a diagram illustrating the camera coordinate system and the world coordinate system. Figure 12 is a diagram illustrating an example of a flowchart showing the control procedure performed by the information processing unit. Figure 13 is a diagram illustrating an example of guide information generated by a modified guide information generation unit. Figure 14 is a diagram illustrating an example of guide information generated by a modified guide information generation unit. Figure 15 is a diagram illustrating an example of guide information generated by a modified guide information generation unit. Figure 16 shows an example of guide information generated by the modified example guide information generation unit. Figure 17 shows an example of a flowchart showing the control procedure performed by the modified example information processing unit.

[0009] The information processing method, information processing apparatus, and program according to the embodiments of this disclosure will be described in detail below with reference to the attached drawings.

[0010] Figure 1 is a diagram showing an example of the configuration of the information processing system 1 according to this embodiment.

[0011] As shown in Figure 1, the information processing system 1 comprises a server 100, a vehicle 200, and an operator terminal 300. The operator terminal 300 is connected to the server 100 and the vehicle 200 via a network such as the Internet. The connection between the vehicle 200 and the operator terminal 300 may also be made via the server 100. The vehicle 200 in this embodiment has two driving modes: a first mode indicating a state in which autonomous driving is possible, and a second mode indicating a state in which driving is possible in response to operations received by the operator terminal 300, and can be driven by switching between the first mode and the second mode. The number of vehicles 200 included in the information processing system 1 can be arbitrarily changed according to design conditions, etc., and it is sufficient for the information processing system 1 to include one or more vehicles 200.

[0012] First, let's explain the configuration of the server 100. The server 100 is a device that stores pre-configured individual information for each vehicle 200, which is necessary for generating guide information indicating the direction of travel of the vehicle 200, and transmits this individual information to the operator terminal 300. As shown in Figure 1, the server 100 has a storage unit 110 and an individual information distribution unit 120.

[0013] The memory unit 110 stores individual information about the vehicle 200. Figure 2 shows an example of the data structure of the individual information. As shown in Figure 2, the individual information includes vehicle ID (identification), camera information, guide generation information, vehicle body information, and display image information.

[0014] The vehicle ID is identification information for vehicle 200 and identifies vehicle 200 included in the information processing system 1.

[0015] Camera information refers to information about the camera mounted on the vehicle 200. Camera information includes, for example, the camera ID, internal parameters, external parameters, and the camera's mounting position. The camera ID is the camera's identification information; internal parameters are parameters representing the camera's characteristics, such as focal length and lens distortion coefficient; and external parameters are parameters representing the camera's position and orientation in three-dimensional space. These lens parameters can be determined by known calibration methods. For example, as shown in Figure 3, a calibration board 400, whose position in the world coordinate system is known, can be imaged with a camera, and the camera's internal and external parameters can be calculated based on the captured image.

[0016] Guide generation information is information used to generate the guide information described later, and includes guide type information indicating the type of guide information, and display parameters defined for each guide type. As shown in Figure 4, for example, there are planar type, vertical line type, horizontal line type, and mark type guide information.

[0017] Figure 5 is a diagram illustrating the display parameters corresponding to the types of planar guide information. The display parameters corresponding to the types of planar guide information include parameters that specify the "guide width," "guide start position," "guide end position," "guide color," "guide transparency," "guide thickness," and "whether or not the guide is filled."

[0018] Furthermore, the guide information may include scales that indicate the unit quantities used when dividing the guide information into multiple sections. Display parameters corresponding to the type of planar guide information may also include parameters for specifying the "type of scale," "interval of scale," etc. The "type of scale" mentioned above refers to the type of method for setting the scale as the unit quantity when dividing the guide information into multiple sections. Examples include a "time-based scale" set based on the distance traveled by the vehicle 200 in a unit of time, or a "distance-based scale" set based on a unit length regardless of the speed of the vehicle 200.

[0019] Figure 6 is a diagram illustrating the display parameters corresponding to the types of vertical line guide information. The display parameters corresponding to the types of vertical line guide information include parameters that specify the "start position of the guide," the "end position of the guide," the "color of the guide," the "transparency of the guide," and the "thickness of the guide."

[0020] Figure 7 is a diagram illustrating the display parameters corresponding to the types of horizontal line guide information. The display parameters corresponding to the types of horizontal line guide information include parameters that specify "guide width," "guide distance," "guide color," "guide transparency," and "guide thickness."

[0021] Figure 8 is a diagram illustrating the display parameters corresponding to the types of mark-shaped guide information. The display parameters corresponding to the types of mark-shaped guide information include parameters that specify "position of the mark on the left," "position of the mark on the right," "shape of the mark," "color of the guide," and "transparency of the guide."

[0022] Let's return to Figure 2 and continue the explanation. Vehicle information includes external dimensions such as the vehicle's width and vehicle performance information such as the wheelbase. Display video information is video information displayed to the operator when the vehicle 200 is remotely operated or remotely monitored by the operator, and includes information such as whether or not to crop the camera's captured video for display. Note that the information stored in the storage unit 110 is not limited to this.

[0023] Returning to Figure 1, the explanation continues. The individual information distribution unit 120 transmits the individual information stored in the storage unit 110 to the operator terminal 300. Regarding the guide generation information included in the individual information, the unit may transmit predetermined guide types and display parameters, or it may transmit guide types and display parameters selected by the user.

[0024] In the following description of this embodiment, the individual information distribution unit 120 will be described using the example of transmitting guide generation information corresponding to the guide type of the display format on the plane.

[0025] The server 100 is a computer device that includes at least a processor and memory for storing various types of information such as programs. The storage unit 110 is implemented by memory, and the individual information distribution unit 120 can be implemented by the processor executing a program stored in memory. However, the server is not limited to this configuration, and the individual information distribution unit 120 may be implemented by a dedicated hardware circuit.

[0026] Next, the configuration of the vehicle 200 will be described. The vehicle 200 is an example of a "mobile body" that can be driven autonomously or remotely operated by an operator. As shown in Figure 1, the vehicle 200 includes a control unit 210 and a camera 220.

[0027] The control unit 210 comprehensively controls the operation of the vehicle 200. The control unit 210 is a computer device that includes at least a processor and a memory for storing various information such as programs. The camera 220 is a camera that photographs the surroundings, including the direction of travel of the vehicle 200, and obtains captured video data. The captured video data will be described as captured video.

[0028] Furthermore, the control unit 210 is connected to an in-vehicle network such as a CAN (Controller Area Network), and is connected to internal and external sensors mounted on the vehicle 200 via the in-vehicle network.

[0029] The internal environment sensors are sensors that obtain observational information about the internal environment of the vehicle 200. The observational information includes information such as the vehicle 200's position, speed, acceleration, steering angle, steering direction, accelerator pedal depression angle, and vibration. Specifically, the internal environment sensors include position sensors (GNSS (Global Navigation Satellite System), GPS (Global Positioning System)). The internal environment sensors also include, for example, an inertial measurement unit (IMU), acceleration sensors, speed sensors, gyro sensors and other speed sensors, a rotary encoder, a steering angle sensor, a steering direction sensor, an accelerator pedal depression angle detection sensor, a vibration sensor, etc. Furthermore, the internal environment sensors also include sensors for detecting mode information indicating whether the vehicle 200 is autonomously driving or not, and sensors for detecting whether the vehicle 200 is stopped or not.

[0030] The external sensor is a sensor that obtains distance information. The distance information indicates the distance from vehicle 200 to objects in the vicinity of vehicle 200.

[0031] External sensors include, for example, distance sensors. Distance sensors include, for example, millimeter-wave radar, laser sensors, and distance image sensors. Laser sensors include, for example, two-dimensional LiDAR (Laser imaging Detection and Ranging) sensors and three-dimensional LiDAR sensors installed parallel to the horizontal plane.

[0032] Next, the functions of the control unit 210 will be described. As shown in Figure 1, the control unit 210 includes a vehicle data acquisition unit 211, a vehicle data distribution unit 212, a video data acquisition unit 213, and a video data distribution unit 214. In the example in Figure 1, only the functions necessary for explaining the main parts of this embodiment are illustrated, but the functions of the control unit 210 are not limited to these. In this embodiment, the processor executes a program stored in memory to realize the functions of each of the above-mentioned parts. However, it is not limited to this, and some or all of these functions may be realized by dedicated hardware circuits (semiconductor integrated circuits, etc.).

[0033] The vehicle data acquisition unit 211 acquires vehicle data from the in-vehicle network. The vehicle data distribution unit 212 transmits the vehicle data acquired by the vehicle data acquisition unit 211 to the operator terminal 300.

[0034] Here, vehicle data refers to information about the driving environment of vehicle 200. Vehicle data includes, for example, the position, speed, acceleration, steering angle, steering direction, size, number and distance to surrounding objects, accelerator pedal depression angle, vibration, mode information indicating whether vehicle 200 is driving autonomously or not, and information indicating whether vehicle 200 is stopped or not.

[0035] The video data acquisition unit 213 acquires the captured video footage taken by the camera 220. The video data acquisition unit 213 also derives analysis results by analyzing the captured video footage obtained by the camera 220 using a known image processing method. The analysis results include information about the surrounding environment, such as the brightness around the vehicle 200.

[0036] The video data distribution unit 214 transmits the captured video acquired by the video data acquisition unit 213 and the derived analysis results to the operator terminal 300.

[0037] Next, the configuration of the operator terminal 300 will be described. The operator terminal 300 is an example of an "information processing device" and is used for remote control and monitoring of the vehicle 200 by an operator. As shown in Figure 1, the operator terminal 300 comprises an information processing unit 310 and a display unit 320. The information processing unit 310 is composed of a computer device that includes at least a processor and a memory for storing various types of information such as programs. The display unit 320 is a device that displays various types of information and is composed of, for example, a liquid crystal display device.

[0038] As shown in Figure 1, the information processing unit 310 includes an individual information acquisition unit 311, a vehicle data acquisition unit 312, a video data acquisition unit 313, a guide information generation unit 314, a superimposed video generation unit 315, and a display control unit 316. In the example in Figure 1, only the functions necessary for explaining the main parts of this embodiment are illustrated, but the functions of the information processing unit 310 are not limited to these. In this embodiment, the processor realizes the functions of each of the above-mentioned parts by executing a program stored in memory. However, it is not limited to this, and some or all of these functions may be realized by dedicated hardware circuits (semiconductor integrated circuits, etc.).

[0039] The individual information acquisition unit 311 acquires individual information about the vehicle 200 via the individual information distribution unit 120. The vehicle data acquisition unit 312 acquires vehicle data of the vehicle 200 via the vehicle data distribution unit 212. The video data acquisition unit 313 acquires captured video and the analysis results of the captured video analyzed by the video data acquisition unit 213 from the vehicle 200 via the video data distribution unit 214.

[0040] The guide information generation unit 314 generates guide information indicating the direction of travel of the vehicle (moving object) according to the state of the vehicle. More specifically, the guide information generation unit 314 generates guide information indicating the direction of travel of the vehicle 200 based on the guide type information acquired by the individual information acquisition unit 311, display parameters defined for each guide type, vehicle body information, etc., and the vehicle data acquisition unit 312 acquired the speed and steering angle of the vehicle 200, etc. For example, the width of the guide information is determined according to the width of the vehicle 200, the direction in which the guide information extends is determined according to the steering angle of the vehicle 200, and the guide information is generated.

[0041] Furthermore, the above-mentioned guide information includes "text information," such as numbers indicating unit quantities when dividing the guide information into multiple parts. The guide information generation unit 314 places the text information included in the guide information, for example, on the side of the guide information.

[0042] Then, the guide information generating unit 314 changes the position of the text information included in the guide information based on the display mode of the generated guide information. In the present embodiment, the guide information generating unit 314 changes the position of the text information in accordance with the traveling direction indicated by the guide information. More specifically, as shown in FIG. 9, when the traveling direction indicated by the guide information is a direction turning left, the text information is arranged on the right side of the guide information, and when the traveling direction indicated by the guide information is a direction turning right, the text information is arranged on the left side of the guide information.

[0043] Further, the guide information generating unit 314 generates guide information in a monitor coordinate system based on an installation position of a camera and lens parameters of the camera. More specifically, the guide information generating unit 314 converts the guide information in a world coordinate system into guide information in a camera coordinate system with the installation position of the camera 220 as a reference, and converts the converted guide information into a two-dimensional monitor coordinate system.

[0044] Here, the camera coordinate system and the monitor coordinate system will be described. As coordinate systems for representing guide information, there are a two-dimensional monitor coordinate system as shown in FIG. 10, a three-dimensional camera coordinate system based on the installation position of a camera (camera 220) as shown in FIG. 11, and a three-dimensional world coordinate system that does not depend on the installation position of the camera. The monitor coordinate system is a coordinate system on the display unit 320, and the world coordinate system is an absolute coordinate.

[0045] The guide information generating unit 314 converts guide information in the world coordinate system into a camera coordinate system based on the installation position of the camera (camera 220) installed in the vehicle 200, with reference to the installation position of the camera (camera 220), further converts the converted guide information into the monitor coordinate system based on the lens parameters of the camera (camera 220), and generates guide information in the monitor coordinate system.

[0046] The superimposed video generating unit 315 generates a superimposed video by superimposing the guide information in the monitor coordinate system generated by the guide information generating unit 314 onto the captured video acquired by the video data acquiring unit 313.

[0047] The control unit 316 controls the operator terminal 300 such that the superimposed image generated by the superimposed image generation unit 315, which includes the above guide information, is displayed on the display unit 320. Further, a part of the guide information or text information may be selected and highlighted by an operator's selection operation. The operator can perform remote control such as remote monitoring and remote operation of the vehicle 200 by visually recognizing the information displayed on the display unit 320.

[0048] FIG. 12 is an example of a flowchart showing the above-described control procedure performed by the operator terminal 300.

[0049] First, when remote control such as remote monitoring and remote operation of the vehicle 200 is started by an operator, the individual information acquisition unit 311 acquires information such as the installation position of the camera 220 and lens parameters as individual information related to the vehicle 200 via the individual information distribution unit 120 (step S101).

[0050] Next, the vehicle data acquisition unit 312 acquires the position, steering angle information, and speed of the vehicle 200 via the vehicle data distribution unit 212 (step S102).

[0051] Next, the individual information acquisition unit 311 acquires vehicle body information such as the outer dimensions and wheelbase of the vehicle 200 as individual information related to the vehicle 200 via the individual information distribution unit 120 (step S103), and further acquires guide types set for the vehicle 200 and parameter information corresponding to the guide types (step S104).

[0052] The guide information generation unit 314 generates guide information in the world coordinate system based on each piece of information acquired in steps S101 to S104 (step S105).

[0053] Subsequently, the guide information generation unit 314 converts the guide information into the camera coordinate system based on the individual information and vehicle data (step S106), and then converts it into the monitor coordinate system (step S107).

[0054] The superimposed image generation unit 315 generates a superimposed image by superimposing the captured video acquired by the video data acquisition unit 313 and the guide information converted to the monitor coordinate system, and the display control unit 316 controls the display unit 320 to display the superimposed image (step S108).

[0055] As described above, the information processing method executed by the operator terminal 300 according to this embodiment generates guide information indicating the direction of travel of the vehicle 200 according to the state of the vehicle 200, changes the position of the text information included in the guide information based on the display mode of the generated guide information, and displays the guide information including the text information whose position has been changed. More specifically, the operator terminal 300 controls the display unit 320 to display the text information on the right side of the guide information if the direction of travel indicated by the guide information is a left turn, and to display the text information on the left side of the guide information if the direction of travel indicated by the guide information is a right turn. Since the display space is narrow on the inside of the turning direction, it may not be possible to display the text information, so by placing the text information on the opposite side of the turning direction, it is possible to suppress the loss of visibility of the text information (guide information).

[0056] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined.

[0057] Furthermore, the effects described in the embodiments of this specification are merely illustrative and not limiting, and other effects may also occur.

[0058] The following are some variations.

[0059] (1) Modification 1 In this embodiment, the operator terminal 300 changes the position of the text information included in the guide information based on the display mode of the guide information, and in the control that displays the guide information including the text information whose position has been changed on the display unit 320, the position of the text information is changed based on the "direction of travel indicated by the guide display" among the display modes of the guide information, but the position of the text information may be changed based on information from a different display mode.

[0060] For example, the guide information may include information indicating whether or not the guide information will overflow when the guide information generation unit 314 performs control to convert the guide information to the monitor coordinate system, and the guide information generation unit 314 may change the position of the text information included in the guide information based on the status of whether or not the guide information will overflow.

[0061] In this configuration, the operator terminal 300 controls the guide information to place the text information on the side of the guide information that does not extend beyond the display screen if one side of the guide information extends beyond the display screen.

[0062] The above information regarding "whether or not the guide information protrudes" includes not only information on whether the coordinate values ​​of the side of the guide information are such that they fit within the display screen of the display unit 320, but also information indicating which side of the guide information (left or right) is not fitting within the display screen if it does not.

[0063] The guide information generation unit 314 refers to the status of "whether or not the guide information is overflowing" and, if it determines that there is overflowing guide information (the side of the guide information does not fit on the display screen), it generates guide information such that if the right side of the guide information does not fit on the display screen, it places text information on the left side of the guide information, and if the left side of the guide information does not fit on the display screen, it places text information on the right side of the guide information. The display control unit 316 then controls the display unit 320 to display the guide information.

[0064] (2) Modification 2 In this embodiment, the operator terminal 300 generates guide information indicating the direction of travel of the vehicle 200 according to the state of the vehicle 200. However, if the vehicle 200 has multiple driving modes, the operator terminal 300 may provide different guide information for each driving mode.

[0065] Specifically, for example, when a vehicle 200 appropriately switches between a first mode indicating a state in which it is capable of autonomous driving and a second mode indicating a state in which it is capable of driving in response to an operation received by an operator terminal 300, the guide information generation unit 314 may refer to mode information indicating whether or not the vehicle 200 is in autonomous driving mode from the vehicle data acquired by the vehicle data acquisition unit 312, and generate guide information such that the size of the guide information corresponding to the first mode is larger than the size of the guide information corresponding to the second mode.

[0066] Furthermore, for example, when the guide information generation unit 314 generates guide information corresponding to the second mode, it may generate the guide information based on the operating angle information of a controller such as a steering wheel used by the operator, instead of the steering angle information among the vehicle data acquired by the vehicle data acquisition unit 312, or it may generate the guide information based on both the steering angle information of the vehicle 200 and the operating angle information of the controller.

[0067] Furthermore, for example, the guide information generation unit 314 may generate guide information such that the color of the guide information corresponding to the first mode is different from the color of the guide information corresponding to the second mode.

[0068] (3) Modification 3 As a modification, the operator terminal 300 may change the guide information according to the information around the vehicle 200.

[0069] For example, the operator terminal 300 may change the display mode of the guide information according to the positional relationship between the vehicle 200 and obstacles (surrounding objects).

[0070] Specifically, first, the guide information generation unit 314 refers to the size, number, and distance to surrounding objects from the vehicle data acquired by the vehicle data acquisition unit 312, converts the coordinates of the objects into the monitor coordinate system, and the superimposed image generation unit 315 superimposes the images of the objects onto the captured video.

[0071] As shown in Figure 13, the display control unit 316 controls the display unit 320 to display the portion of the guide information that overlaps with the surrounding object as semi-transparent when the generated guide information overlaps with the image of the object. Furthermore, when the text information of the generated guide information overlaps with the image of the object, the display unit 320 controls the display unit 320 so as not to display the portion of the text information that overlaps with the surrounding object.

[0072] In addition to the above control, the display unit 320 may also be controlled to change the color of the guide information when the generated guide information overlaps with the image of the object.

[0073] Furthermore, for example, the operator terminal 300 may change the guide information according to the degree of congestion on the road on which the vehicle 200 is traveling.

[0074] Specifically, as shown in Figure 14, the guide information generation unit 314 refers to the number of surrounding objects in the vehicle data acquired by the vehicle data acquisition unit 312. If the number of surrounding objects is higher than a threshold, it determines that the road on which the vehicle 200 is traveling is congested and changes the guide information to be smaller, as shown in Figure 14(a). If the number of surrounding objects is lower than a threshold, it determines that the road on which the vehicle 200 is traveling is not congested and changes the guide information to be larger, as shown in Figure 14(b).

[0075] Alternatively, for example, the operator terminal 300 may change the guide information according to the maximum speed limit set for the road on which the vehicle 200 is traveling.

[0076] Specifically, as shown in Figure 15, the guide information generation unit 314 refers to the location information of the vehicle 200 from the vehicle data acquired by the vehicle data acquisition unit 312. If the vehicle 200 is traveling on a road with a low maximum speed limit, the unit amount used to divide the guide information into multiple sections is reduced, as shown in Figure 15(a), and the distance between the end and start positions of the guide is reduced. If the vehicle 200 is traveling on a road with a high maximum speed limit, the unit amount used to divide the guide information into multiple sections is increased, as shown in Figure 15(b), and the distance between the end and start positions of the guide is increased.

[0077] Furthermore, for example, the operator terminal 300 may change the color or brightness of the guide information according to the ambient light surrounding the vehicle 200.

[0078] Specifically, as shown in Figure 16, the guide information generation unit 314 refers to the ambient brightness information of the vehicle 200 from the analysis results of the captured video acquired from the vehicle 200 by the video data acquisition unit 313. If the ambient brightness is lower than a threshold, the unit changes the color or brightness of the guide information to make it brighter, as shown in Figure 16(a). If the ambient brightness is higher than a threshold, the unit changes the color or brightness of the guide information to make it darker, as shown in Figure 16(b).

[0079] (4) Modification 4 The vehicle 200 according to this embodiment may be equipped with multiple cameras 220, and the operator terminal 300 may display guide information superimposed on the captured video footage taken by the cameras 220 provided for each direction of movement of the vehicle 200.

[0080] For example, in a configuration in which a vehicle 200 capable of moving forward and backward is equipped with a front camera to photograph the front of the vehicle 200 and a rear camera to photograph the rear of the vehicle 200, the operator terminal 300 may include a determination unit that determines whether the vehicle 200 is moving forward or backward, and may display guide information superimposed on the image captured by the front camera when the vehicle 200 is moving forward, and display guide information superimposed on the image captured by the rear camera when the vehicle is moving backward.

[0081] Figure 17 is an example of a flowchart showing the control procedure performed by the operator terminal 300 in the modified example described above.

[0082] First, when remote control, such as remote monitoring and remote operation of the vehicle 200, is initiated by the operator terminal 300, the determination unit refers to the vehicle data acquired by the vehicle data acquisition unit 312 and determines whether or not the vehicle 200 is performing a reverse operation (step S201).

[0083] If the vehicle 200 is moving in reverse (step S201: Yes), the individual information acquisition unit 311 acquires information such as the installation position and lens parameters of the rear camera of the vehicle 200 as individual information about the vehicle 200 via the individual information distribution unit 120 (step S202).

[0084] From here on, the processing contents of steps S203 to S209 are the same as the processing contents of steps S102 to S108 described in the above embodiment, so the explanation will be omitted. After step S209, the determination unit performs step S201, in which it refers to the vehicle data acquired by the vehicle data acquisition unit 312 and determines whether or not the vehicle 200 is performing a reverse operation.

[0085] On the other hand, if the vehicle 200 is not moving in reverse (step S201: No), the individual information acquisition unit 311 acquires information such as the installation position and lens parameters of the vehicle 200's front camera as individual information about the vehicle 200 via the individual information distribution unit 120 (step S210).

[0086] From step S210 onward, the procedures performed by the operator terminal 300 in steps S211 to S217 are the same as the procedures performed on the rear camera or the video footage captured by the rear camera in each of the procedures S203 to S209 described above, but applied to the video footage captured by the front camera or the rear camera, so their explanation will be omitted.

[0087] (5) Modification 5 The information processing system 1 of this embodiment, which includes the operator terminal 300, includes a plurality of vehicles 200, and the operator terminal 300 generates guide information according to the status of each vehicle 200. Here, when adding another vehicle to the information processing system 1, in order to estimate the installation position and lens parameters of the camera equipped on the added vehicle, the measurement standards may be calibrated in advance before the start of actual operation, and the camera installation position and lens parameters estimated in the calibration may be stored in the storage unit 110 of the server 100 in association with the recognition ID of the added vehicle.

[0088] Furthermore, when the operator terminal 300 generates guide information for the added vehicle 200 and displays it on the display unit 320, it may refer to the individual vehicle information and vehicle data of the added vehicle 200 and propose to the operator the type of guide to display based on pre-set rules.

[0089] Although embodiments and modifications of the present invention have been described above, the present invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments described above. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0090] Furthermore, the program executed by the operator terminal 300 according to this embodiment is provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk).

[0091] Furthermore, the program executed on the operator terminal 300 of this embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Alternatively, the program executed on the operator terminal 300 of this embodiment may be provided or distributed via a network such as the Internet.

[0092] 1 Information Processing System 100 Server 110 Storage Unit 120 Individual Information Distribution Unit 200 Vehicle 210 Control Unit 211 Vehicle Data Acquisition Unit 212 Vehicle Data Distribution Unit 213 Video Data Acquisition Unit 214 Video Data Distribution Unit 220 Camera 300 Operator Terminal 310 Information Processing Unit 311 Individual Information Acquisition Unit 312 Vehicle Data Acquisition Unit 313 Video Data Acquisition Unit 314 Guide Information Generation Unit 315 Superimposed Video Generation Unit 316 Display Control Unit 320 Display Unit 400 Calibration Board

Claims

1. An information processing method performed by an information processing device, comprising: generating guide information indicating the direction of movement of a moving object according to the state of the moving object; changing the position of text information included in the guide information based on the display configuration of the generated guide information; and displaying the guide information including the text information whose position has been changed.

2. The information processing method according to claim 1, wherein the position of the text information is changed according to the direction of travel indicated by the guide information.

3. The information processing method according to claim 2, wherein if the direction of travel indicated by the guide information is a turn to the left, the text information is placed to the right of the guide information, and if the direction of travel indicated by the guide information is a turn to the right, the text information is placed to the left of the guide information.

4. The information processing method according to claim 1, wherein if one side of the guide information extends beyond the display screen, the text information is placed on the side of the guide information that does not extend beyond the display screen.

5. The information processing method according to claim 1, wherein the mobile body has a plurality of operating modes, and the guide information is different for each operating mode.

6. The information processing method according to claim 5, wherein the plurality of driving modes include a first mode indicating a state in which the mobile body is autonomously driven and a second mode indicating a state in which the mobile body is driven by remote control, and the size of the guide information corresponding to the first mode is larger than the size of the guide information corresponding to the second mode.

7. The information processing method according to claim 1, wherein the guide information is changed according to information about the surroundings of the moving body.

8. The information processing method according to claim 7, wherein the display mode of the guide information is changed according to the positional relationship between the moving body and the obstacle.

9. The information processing method according to claim 7, wherein the guide information is changed according to the degree of congestion on the road on which the moving object is traveling.

10. The information processing method according to claim 9, wherein the guide information is modified so as to be reduced when the degree of congestion on the road on which the moving object is traveling is high, and increased when the degree of congestion is low.

11. The information processing method according to claim 7, wherein the color or brightness of the guide information is changed according to the brightness surrounding the moving body.

12. The information processing method according to claim 1, wherein the moving body is equipped with a camera that photographs the direction of travel of the moving body, and the captured video showing the image captured by the camera and the guide information are superimposed and displayed.

13. The information processing method according to claim 12, which generates the guide information to be superimposed on the captured image based on the installation position of the camera and the lens parameters of the camera.

14. An information processing device comprising: a guide information generation unit that generates guide information indicating the direction of travel of a moving object according to the state of the moving object; a modification unit that changes the position of text information included in the guide information based on the display mode of the guide information generated by the guide information generation unit; and a display control unit that performs control to display the guide information, including the text information whose position has been changed by the modification unit, on a display unit.

15. A program for causing an information processing device to execute: a guide information generation step of generating guide information indicating the direction of travel of a moving object according to the state of the moving object; a modification step of changing the position of text information included in the guide information based on the display configuration of the guide information generated by the guide information generation step; and a display control step of controlling the display unit to display the guide information, including the text information whose position has been changed by the modification step.

Images