Information processing method, information processing device, and program

Abstract

[Problem] To enable imaging in which a prescribed subject is included within an angle of view as appropriate by a movable imaging device, and further enhance the experience of the subject. [Solution] An information processing method executed by a processor and comprising: performing imaging control of one or more movable imaging devices that capture images of one or more subjects; and generating, on the basis of unique information relating to the one or more subjects, imaging control information and a movement path that enable imaging in which the one or more subjects are included within an angle of view, and performing movement control of the one or more movable imaging devices.

Description

Information processing method, information processing apparatus, and program

[0001] The present disclosure relates to an information processing method, an information processing apparatus, and a program.

[0002] In recent years, imaging is not limited to imaging by ground cameras at sports venues and the like; imaging by drones is also performed, and each captured image is appropriately distributed.

[0003] Further, Patent Document 1 below discloses a technique of capturing an image of a spectator in a specific seat by a camera installed at a sports venue, and processing the faces of spectators other than the specific spectator in the captured image so that they cannot be identified.

[0004] Japanese Patent Application Laid-Open No. 2023-5710

[0005] However, the technique described in Patent Document 1 relates to imaging by a fixed camera installed at a sports venue for imaging spectator seats, and does not consider imaging control by a mobile imaging device.

[0006] Accordingly, the present disclosure proposes an information processing method, an information processing apparatus, and a program that enable a mobile imaging device to capture an image of a predetermined subject appropriately included in an angle of view, and further enhance the experience of the subject.

[0007] According to the present disclosure, there is provided an information processing method including: a processor performing imaging control of one or more movable mobile imaging devices that image one or more subjects; and generating imaging control information and a movement route that enable imaging including the one or more subjects in an angle of view based on unique information related to the subjects, and performing movement control of the one or more mobile imaging devices.

[0008] Further, according to the present disclosure, there is provided an information processing apparatus including a control unit that performs imaging control of one or more movable mobile imaging devices that image one or more subjects, wherein the control unit further generates imaging control information and a movement route that enable imaging including the one or more subjects in an angle of view based on unique information related to the subjects, and performs movement control of the one or more mobile imaging devices.

[0009] Furthermore, according to this disclosure, a computer is provided as a control unit that controls the imaging of one or more movable mobile imaging devices that image one or more subjects, and the control unit further generates imaging control information and movement paths that enable imaging in which one or more subjects are included in the field of view, based on specific information relating to the subjects, and controls the movement of the one or more mobile imaging devices.

[0010] This is a diagram illustrating the outline of an information processing system according to one embodiment of the present disclosure. This is a diagram showing an example of the overall configuration of the information processing system 1 according to this embodiment. This is a block diagram showing an example of the configuration of the drone 10 according to this embodiment. This is a block diagram showing an example of the configuration of the server 20 according to this embodiment. This is a sequence diagram showing an example of the operation processing flow of the information processing system 1 according to this embodiment. This is a flowchart showing the flow of movement and imaging control of the drone 10 by the server 20 according to this embodiment. This is a diagram illustrating an example of determining the subject field of view according to subject ratio information according to this embodiment. This is a flowchart showing the modification process of movement path and imaging control information according to this embodiment. This is a diagram showing a part of each movement path of multiple drones according to this embodiment. This is a diagram showing the movement paths of drone 10a and drone 10b in the A section of this embodiment superimposed. This is a flowchart illustrating an example of the video editing process flow according to this embodiment. This is a flowchart illustrating other imaging control processes of the information processing system 1 according to this embodiment. This is a diagram illustrating an example of the imaging position of the drone 10a according to this embodiment. This is a diagram illustrating how to handle situations where imaging is difficult according to this embodiment. This is a diagram showing an example of the frontal arrangement of the drone 10 according to this embodiment. This is a diagram illustrating the separation control of the drone 10 according to this embodiment. This is a diagram showing an example of imaging change of the drone 10a according to this embodiment. This is a diagram illustrating imaging by a drone 10 at competition venue F according to a modified version of this embodiment. This is a flowchart illustrating an example of the drone control flow at competition venue according to a modified version of this embodiment. This is a diagram illustrating an example of drone control at competition venue F according to a modified version of this embodiment. This is a diagram illustrating another example of drone control at competition venue F according to a modified version of this embodiment. This is a diagram illustrating another example of drone control at competition venue F according to a modified version of this embodiment. This is a block diagram illustrating an example of the hardware configuration of an information processing device 900 according to one embodiment of this disclosure.

[0011] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0012] Furthermore, the explanation will be given in the following order: 1. Overview 2. Configuration 2-1. Configuration of Drone 10 2-2. Configuration of Server 20 3. Operation Processing 3-1. Overall Flow 3-2. Control of Movement and Image Capture 3-3. Video Editing Processing 4. Other 4-1. Estimation of the Position of a Specific Spectator U and Estimation of Related Parties 4-2. Other Examples of Image Capture Control 5. Variations 5-1. Operation Processing 5-2. Image Capture Including Spectator T 5-3. Video Editing 6. Hardware Configuration 7. Supplementary Information

[0013] <1. Overview> As one embodiment of this disclosure, an information processing system is described that enables imaging at an event venue using one or more mobile imaging devices to appropriately include one or more predetermined subjects in the field of view. The subjects are at least one of the performers who perform at the event and at least one of the audience members who watch the event. The performance includes at least one of the following: a show, acting, singing, playing an instrument, and a competition, and the performer includes at least one of the following: a performer, an instrumentalist, and a competitor.

[0014] Figure 1 is a diagram illustrating the overview of an information processing system according to one embodiment of the present disclosure. In this embodiment, a concert is given as an example of an event. As shown in Figure 1, there is a performer P (an example of a performer) on the stage S of the concert venue E, who performs music, singing, etc. Also, there are many spectators in the audience area of ​​the concert venue E.

[0015] In this embodiment, a drone 10, such as an unmanned aerial vehicle (UAV), is used as an example of a mobile imaging device. The drone 10 is equipped with a camera (imaging unit). The drone 10 moves around the concert venue E and takes images of performers and audience members as appropriate. The audience includes a specific audience member U. In this embodiment, the drone 10 takes images that include the specific audience member U and performer P in its field of view, and can provide images (videos, still images) for the specific audience member U. The specific audience member U can obtain images of themselves and their favorite performer P together during the concert, enriching their event experience.

[0016] The drone 10 can perform imaging that includes a specific spectator U in its field of view, imaging that includes a performer P, and imaging that includes both the specific spectator U and the performer P in its field of view. The specific spectator U may be a user registered as a contractor of the drone 10.

[0017] Figure 2 is a diagram showing an example of the overall configuration of the information processing system 1 according to this embodiment. As shown in Figure 2, the information processing system 1 includes a mobile imaging device, a drone 10; a remote controller 15 for the drone 10; an operator terminal 30 used by the event management side; a ground camera 32 installed at the event venue; an audio system 34 into which the voice of performer P and the like are input; a server 20; and a user terminal 40.

[0018] The configuration shown in Figure 2 is just one example, and the information processing system 1 does not necessarily have to include all of the components shown in Figure 2. The drone 10 may be manually operated by the remote controller 15, controlled by the server 20, or controlled by the operator terminal 30. The drone 10 may also move (fly) autonomously.

[0019] The operator terminal 30 can be operated by the event organizer. For example, a list of songs to be played at the concert, song information, performance information, etc., can be entered into the operator terminal 30.

[0020] The ground camera 32 is intended to be a camera other than the drone 10 that captures images of the concert venue. The ground camera 32 could take various forms, such as a shoulder camera carried by staff, a fixed camera whose direction and other functions are remotely controlled, or a camera mounted on a crane. The ground camera 32 will primarily capture images of the performer P. There may be multiple ground cameras 32.

[0021] The sound system 34 includes speakers installed in the concert venue, a microphone used by performer P, a microphone for acquiring ambient sounds such as cheers, and an audio device for processing audio signals.

[0022] Server 20 may perform subscriber registration processing for users (audience members U), may have a function to control the drone 10, or may have a function to edit images captured by the drone 10 and the ground camera 32. Server 20 may communicate with the user terminal 40 and obtain information necessary for subscriber registration processing, drone 10 control, and image editing from the user terminal 40. Server 20 may be located at the concert venue or on a network. Furthermore, server 20 may be implemented as a system consisting of multiple devices.

[0023] The user terminal 40 is an information processing terminal operated by the user. The user terminal 40 can be implemented as a smartphone, tablet, HMD (Head Mounted Display), or PC (Personal Computer), etc. In this embodiment, it is assumed that the user is a specific spectator U. Spectator U can operate the user terminal 40 to register as a subscriber to the drone 10 and input necessary information.

[0024] <2. Configuration> Next, the configurations of the drone 10 and server 20 included in the information processing system 1 according to this embodiment will be described with reference to the drawings. The control of the drone 10 may be performed by autonomous control of the drone 10, or by the server 20, remote controller 15, or operator terminal 30. Control of the drone 10 may include the generation of a movement path for the drone 10.

[0025] <<2-1. Configuration of Drone 10>> Figure 3 is a block diagram showing an example of the configuration of the drone 10 according to this embodiment. As shown in Figure 3, the drone 10 includes a communication unit 110, an imaging unit 120, a control unit 130, a sensor unit 140, a storage unit 150, and a drive unit 160.

[0026] (Communication Unit 110) The communication unit 110 has a transmitting unit that transmits data to an external device and a receiving unit that receives data from an external device. The communication unit 110 according to this embodiment communicates with an external device or the Internet using, for example, a wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network, etc.

[0027] For example, the communication unit 110 transmits the captured image to the server 20 in accordance with the control unit 130.

[0028] (Imaging Unit 120) The imaging unit 120 has one or more lenses (optical system) and an image sensor consisting of a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and performs imaging according to the control of the control unit 130.

[0029] (Sensor unit 140) The sensor unit 140 is an acquisition unit that acquires various sensing data. Specifically, the sensor unit 140 includes a position sensor, an IMU (Inertial Measurement Unit), a geomagnetic sensor, an ultrasonic sensor, a barometric pressure sensor, etc.

[0030] A position sensor is a receiver that receives data used to calculate position. For example, it can be implemented using a GNSS receiver that receives signals from navigation satellites such as GPS (Global Positioning System), Quasi-Zenith Satellite System, GLONASS, Galileo, and BeiDou. The signals received from the navigation satellites by the position sensor are output to the control unit 130.

[0031] The IMU detects information on the three-dimensional inertial motion of the drone 10 (three-axis acceleration information and three-axis angular velocity information). More specifically, the IMU has an acceleration sensor and an angular velocity sensor, and detects the acceleration and angular velocity of the drone 10 (output of three-axis acceleration information and three-axis angular velocity information). The information detected by the IMU is output to the control unit 130.

[0032] The geomagnetic sensor detects the Earth's magnetic field and determines direction. The geomagnetic sensor is also called an electronic compass. The ultrasonic sensor emits ultrasonic waves and, based on the reflected waves from the direction of emission, detects objects around the drone 10 and calculates the distance to those objects. The barometric pressure sensor detects changes in atmospheric pressure and measures the altitude of the drone 10. The information detected by the geomagnetic sensor, ultrasonic sensor, and barometric pressure sensor is output to the control unit 130.

[0033] The above describes a specific example of the sensor unit 140, but the sensor unit 140 is not limited to this and may have other sensors, or it may not have all of the sensors described above.

[0034] (Control Unit 130) The control unit 130 functions as an arithmetic processing unit and control unit, and controls the overall operation of the drone 10 according to various programs. The control unit 130 is implemented by electronic circuits such as a CPU (Central Processing Unit) or a microprocessor. The control unit 130 may also include a ROM (Read Only Memory) for storing programs and calculation parameters to be used, and a RAM (Random Access Memory) for temporarily storing parameters that change as needed.

[0035] The control unit 130 in this embodiment can function as a movement path generation unit 131, a movement control unit 132, an imaging control unit 133, a position and orientation estimation unit 134, and a transmission control unit 135.

[0036] The movement path generation unit 131 generates a movement path for the drone 10, and the generated movement path information is output to the movement control unit 132. The movement path generation unit 131 also generates imaging control information, including information on the imaging direction and field of view along the movement path. The generated imaging control information is transmitted to the imaging control unit 133.

[0037] The movement path generation unit 131 may generate a movement path that passes through specific points. For example, the movement path generation unit 131 generates a movement path and imaging control information based on specific information relating to performer P or a specific audience member U. The specific information can be obtained from an external device via the communication unit 110. For example, the specific information is input from a remote controller 15, operator terminal 30, or server 20.

[0038] Specific information includes location information. If the performer P's position during the concert is predetermined by the production, this position information is acquired as the performer P's location information. The position information may also be input from the operator terminal 30. In the case of a concert where the performer P's position is not predetermined, the movement path generation unit 131 may roughly assume that the center of the stage S is the performer P's position. Note that the performer P's position may change during or between songs. In this case, the performer P's location information may be time-series information based on the progress time of the entire concert or each song.

[0039] The location information of a specific spectator U may be 3D location information based on a seat number. In the case of standing events where the position is not fixed, the location information of a specific spectator U may be 3D location information of the standing area. The 3D location information corresponding to a seat number or standing area can be obtained, for example, from a pre-prepared seat database. The seat database may be located on the server 20 or on the operator terminal 30.

[0040] The movement path generation unit 131 sets predetermined points (3D positions) as waypoints (WP: target points) based on the position information of performer P and specific audience member U. These points include a point from which performer P can be imaged from the front, a point from which specific audience member U can be imaged from the front, and a point from which both performer P and specific audience member U can be included in the field of view. The movement path generation unit 131 then generates a movement path that passes through each WP. In addition, based on the position information of performer P and specific audience member U, the movement path generation unit 131 sets the imaging direction and field of view for each WP and during movement for the imaging unit 120, and generates this as imaging control information. The imaging direction and field of view may be determined by considering a number of pre-set composition patterns.

[0041] Furthermore, the specific information may include subject ratio information indicating the time proportion in which performer P and a specific audience member U are captured during continuous imaging during the event. Subject ratio information refers to information such as the time proportion in which performer P is shown, the time proportion in which a specific audience member U is shown, and the time proportion in which a specific audience member U and performer P are shown together. The time proportion may be indicated by a stepwise index such as "high, medium, low." Subject ratio information may be arbitrarily entered by the user on the user terminal 40. Note that performer P is not limited to one person, but may be multiple people (performers P1 to Pn). If there are multiple performers, such as a group of artists, the user may specify one or more performers P that they like (so-called "favorites") and enter the subject ratio information. Subject ratio information may further include the specification of the field of view or composition when imaging. The user can specify the desired field of view and composition. For example, they may specify a close-up of performer P's face, full body, profile, back view, or the entire venue. The angle of view and composition may be specified by selecting from pre-prepared templates for angle of view and composition. The movement path generation unit 131 may generate the movement path and imaging control information of the drone 10 during the event based on the position information of the performer P and a specific audience member U, and the subject ratio information.

[0042] Further, the unique information also includes a face image of a subject (performer P or specific spectator U), and the face image is used for face recognition. Furthermore, the unique information is not limited to face images, and may also include information used for recognizing the subject, such as the unique identification information of an information processing terminal owned by or attached to the subject.

[0043] The functions of the movement route generation unit 131 included in the drone 10 have been described above. Note that the generation of the movement route is not limited to being performed by the drone 10. The movement route may be generated by an external device such as the remote controller 15, the server 20, or the operator terminal 30, and information of the generated movement route may be transmitted to the drone 10. In addition, the imaging control information may also be generated by the remote controller 15, the server 20, or the operator terminal 30.

[0044] The movement control unit 132 controls the driving unit 160 so that the drone 10 moves according to the movement route acquired from the movement route generation unit 131 or an external device, and executes movement control of the drone 10. More specifically, the movement control unit 132 performs drive control (for example, position control and attitude control) on the driving unit 160 based on information of the position and attitude of the drone 10 estimated by the position and attitude estimation unit 134, so that the drone 10 flies according to the movement route.

[0045] The movement control unit 132 may control the driving unit 160 in accordance with operation control information from the remote controller 15 to execute movement control of the drone 10.

[0046] The movement control unit 132 may control the driving unit 160 to move to a point where the subject can be imaged based on position information of the subject, and execute movement control of the drone 10. For example, the movement control unit 132 may control the driving unit 160 to move to a position where both the performer P and the specific spectator U can be included in the angle of view based on respective pieces of position information thereof, and execute movement control of the drone 10.

[0047] The movement control unit 132 may analyze the captured image captured by the imaging unit 120 to recognize the subject, control the driving unit 160 to automatically track the subject while keeping the subject in focus, and execute movement control of the drone 10.

[0048] The movement control unit 132 may perform each of the movement controls described above with reference to the map information (three-dimensional mapping data) of the concert venue E stored in the storage unit 150.

[0049] The imaging control unit 133 controls the imaging unit 120 in accordance with imaging control information acquired from the movement route generation unit 131 or an external device, and executes imaging of a subject.

[0050] The position and orientation estimation unit 134 estimates the position and orientation of the drone 10 based on information input from the sensor unit 140. For example, the position and orientation estimation unit 134 performs self-position estimation processing based on a signal received by a position sensor from a navigation satellite. Further, the position and orientation estimation unit 134 can perform self-position estimation processing using a technology such as SLAM (Simultaneous Localization and Mapping) or LiDAR even in environments where data from navigation satellites cannot be received, such as indoors. The position and orientation estimation unit 134 may receive signals transmitted from devices installed indoors (such as beacons, Wi-Fi, RFID (Radio Frequency Identification), geomagnetism, UWB (Ultra Wide Band), sound waves, and visible light) to perform self-position estimation processing. Further, the position and orientation estimation unit 134 may use the map information (three-dimensional mapping data) stored in the storage unit 150 when performing the self-position estimation processing. In addition, the position and orientation estimation unit 134 performs orientation estimation processing based on information detected by an IMU and a geomagnetic sensor. The position and orientation estimation unit 134 continuously estimates the position and orientation.

[0051] The transmission control unit 135 performs control to transmit the acquired information from the communication unit 110 to an external device. For example, the transmission control unit 135 performs control to transmit a captured image captured by the imaging unit 120 to the server 20. The transmission control unit 135 may transmit such a captured image to the remote controller 15 in real time. Further, the transmission control unit 135 may transmit information on the self-position and self-orientation estimated by the position and orientation estimation unit 134 to the server 20 or the remote controller 15.

[0052] (Storage Unit 150) The storage unit 150 is implemented by a storage medium that stores programs used in the processing of the control unit 130, calculation parameters, parameters that change as needed, etc.

[0053] The storage unit 150 in this embodiment may store, for example, the movement path, imaging control information, captured images, a history of estimated position and orientation results of the drone 10, and map information.

[0054] (Drive Unit 160) The drive unit 160 is composed of, for example, motors that drive multiple rotors, and the thrust generated by these rotors makes the drone 10 fly.

[0055] The configuration of the drone 10 has been described in detail above. However, the configuration of the drone 10 according to this disclosure is not limited to the example shown in Figure 3. For example, the drone 10 may further have an operation input unit or a display unit. Also, the sensor unit 140 may have various sensors such as an illuminance sensor, a temperature sensor, a wind speed sensor, or a rainfall sensor.

[0056] Furthermore, the drone 10 may also have a microphone (hereinafter referred to as "microphone") as a sensor unit 140. The control unit 130 of the drone 10 may analyze the audio data acquired by the microphone and use the audio analysis results to generate a movement path.

[0057] Furthermore, the drone 10 may not have a movement path generation unit 131. The drone 10 may receive movement path and imaging control information generated by an external device such as a server 20, and perform movement control and imaging control.

[0058] <<2-2. Server 20 Configuration>> Figure 4 is a block diagram showing an example of the configuration of the server 20 according to this embodiment. As shown in Figure 4, the server 20 includes a communication unit 210, a control unit 220, and a storage unit 230.

[0059] (Communication Unit 210) The communication unit 210 has a transmitting unit that transmits data to an external device and a receiving unit that receives data from the external device. The communication unit 210 according to this embodiment may communicate with an external device or the Internet using, for example, a wired or wireless LAN, Wi-Fi, Bluetooth, a mobile communication network, etc.

[0060] Furthermore, the communication unit 210 may receive captured images from the drone 10 and the ground camera 32. The communication unit 210 may also receive a list of songs to be played at the concert, song information, performance information, etc., from the operator terminal 30. The communication unit 210 may also receive audio data from the sound system 34 of the concert venue E (sound of the performance, voices of the performers, and cheers, etc.). The communication unit 210 may also receive information from the user terminal 40 for registering as a subscriber of the drone 10 and information for video editing. Information for subscriber registration may include, for example, the user's location information at the concert venue, such as the seat number. Information for subscriber registration may also include the user's facial image and unique identification information of the user terminal 40 (an example of an information processing terminal).

[0061] (Control Unit 220) The control unit 220 functions as an arithmetic processing unit and control unit, and controls the overall operation within the server 20 according to various programs. The control unit 220 is implemented by electronic circuits such as a CPU or microprocessor. The control unit 220 may also include ROM for storing programs and calculation parameters to be used, and RAM for temporarily storing parameters that change as needed.

[0062] Furthermore, the control unit 220 also functions as a registration processing unit 221, a movement path generation unit 222, a drone control unit 223, and a video editing unit 224.

[0063] The registration processing unit 221 performs registration processing for the user as a subscriber of the drone 10 based on the information transmitted from the user terminal 40. For example, the registration processing unit 221 stores the user's name, the user's seat number at the concert venue E, a facial image, and unique identification information of the user terminal 40 in the storage unit 230 as subscriber information for the drone 10. In this embodiment, the user to be registered as a subscriber (contract user) corresponds to a specific audience member U at the concert venue E. The subscriber information stored in the storage unit 230 is also used as specific information of the specific audience member U.

[0064] The movement path generation unit 222 generates the movement path and imaging control information for the drone 10. Details are the same as those for the movement path generation unit 131 described above, so a detailed explanation is omitted here.

[0065] The drone control unit 223 controls the drone 10 to move and take images according to the movement path and imaging control information generated by the movement path generation unit 222. Specifically, the drone control unit 223 may transmit the movement path and imaging control information generated by the movement path generation unit 222 to the drone 10 from the communication unit 210 and instruct it to perform control according to the movement path and imaging control information. The server 20 may also continuously transmit movement control and imaging control signals to the drone 10 to control its movement and imaging. Furthermore, the server 20 may receive real-time position and attitude information of the drone 10 from the drone 10 and perform real-time movement control and imaging control of the drone 10 based on its current position and attitude and the generated movement path and imaging control information.

[0066] The video editing unit 224 has the function of editing images captured by the drone 10 during the event and generating event video for a specific audience U (i.e., a contracted user). The video editing unit 224 may also generate event video using images captured by one or more ground cameras 32. For example, the video editing unit 224 may generate event video by extracting and stitching together arbitrary scenes from multiple captured images. The video editing unit 224 can also synchronize audio data received from the sound system 34 with the captured images.

[0067] The video editing unit 224 may acquire from the drone 10 not only images of the event, but also the drone 10's movement path and image control history during the event, the performer P's location history, and the location history of a specific audience member U, and use these for video editing. In addition, the video editing unit 224 can acquire facial images of performer P and specific audience member U, and by performing facial recognition on the captured images, it can identify scenes in which performer P and specific audience member U are visible.

[0068] The video editing unit 224 may edit the video according to the video editing information transmitted from the user terminal 40. The video editing information may be subject ratio information included in the specific information described above. In addition, subject ratio information for video editing may be input by the user separately from the subject ratio information at the time of imaging.

[0069] While video editing was described here as an example, the video editing unit 224 may also generate any still image from the captured images of the event. For example, the video editing unit 224 may output a still image that includes performer P and a specific audience member U in its frame.

[0070] (Storage Unit 230) The storage unit 230 is implemented by a storage medium that stores programs used in the processing of the control unit 220, calculation parameters, parameters that change as needed, etc.

[0071] The storage unit 230 in this embodiment stores, for example, generated movement path and imaging control information, various data received from the drone 10, various data received from the ground camera 32, and a registered database of subscriber information.

[0072] The configuration of server 20 has been described in detail above. However, the configuration of server 20 according to this disclosure is not limited to the example shown in Figure 4. For example, server 20 does not necessarily have all the configurations shown in Figure 4. Furthermore, server 20 may be implemented as a system consisting of multiple devices. For example, the movement path generation unit 222 and the video editing unit 224 may be provided on separate servers.

[0073] Furthermore, at least some of the functions of the server 20 may be provided on the drone 10, remote controller 15, operator terminal 30, or user terminal 40. For example, at least one of the movement path generation unit 222 and the drone control unit 223 of the server 20 may be provided on the drone 10, remote controller 15, or operator terminal 30. Also, the video editing unit 224 may be provided on the user terminal 40.

[0074] <3. Operation Processing> <<3-1. Overall Flow>> Figure 5 is a sequence diagram showing an example of the operation processing flow of the information processing system 1 according to this embodiment.

[0075] As shown in Figure 5, first, the user terminal 40 transmits user information (including seat number) and subject ratio information for imaging to the server 20 (step S103). The server 20 then processes the user to register as a subscriber of the drone 10. The registered user information and subject ratio information may also be transmitted to the drone 10 that the user has contracted.

[0076] Next, the operator terminal 30 transmits song information, etc., related to the songs to be performed at the event (in this case, a concert), and performer information, which is information about the performers appearing at the event, to the server 20 (step S106). More specifically, song information, etc., includes a list of songs, detailed song information (including song structure and performance time), etc. In this embodiment, the songs to be performed at the event include not only live performances by performers, but also cases where sound sources are played. Performer information includes the performers' positions, facial images, etc. The song information, etc., and performer information may also be transmitted to the drone 10 contracted by the user.

[0077] Next, the ground camera 32 transmits camera information to the server 20 (step S109). More specifically, the camera information includes the position of the ground camera 32, the direction in which it can capture images, and the field of view in which it can capture images. If the imaging control (i.e., camera work) of the ground camera 32 during the event has been determined, imaging control information may also be transmitted as part of the camera information. The imaging control information for the ground camera 32 may be set by the operator on the operator terminal 30 and transmitted from the operator terminal 30 to the ground camera 32 and the server 20. The imaging control information for the ground camera 32 may also be transmitted to the drone 10 contracted by the user.

[0078] Next, the server 20 uses the registered user information, subject ratio information, song information, performer information, and camera information to generate the movement path and imaging control information of the drone 10 for generating event video for the user (step S112). The user corresponds to a specific audience member U visiting the event venue, and the registered user information corresponds to the specific information of that particular audience member U. The user information also includes the user's location information at the event venue, more specifically, the seat number of the specific audience member U. Details of the generation of the movement path and imaging control information will be described later.

[0079] Next, the server 20 transmits the movement path and imaging control information to the drone 10 (step S115). Note that the generation of the movement path and imaging control information may be performed by the drone 10 (step S116).

[0080] Then, during the event, the drone 10 controls its movement and imaging according to the movement path and imaging control information (step S118). The drone 10 can at least capture images that include specific audience members U and performers P in its field of view.

[0081] Next, the drone 10 transmits the images captured during the event to the server 20 (step S121).

[0082] Meanwhile, the ground camera 32 transmits the images captured during the event to the server 20 (step S124).

[0083] Furthermore, the sound system 34 transmits the audio data acquired during the event to the server 20 (step S127).

[0084] The order of the processes in steps S121 to S127 described above is not particularly limited. Furthermore, the processes in steps S121 to S127 may be performed continuously during the event or after the event. In addition, each piece of information may be transmitted directly to the server 20, transmitted via another device, or input to the server 20 via a storage medium.

[0085] Next, the server 20 performs video editing processing based on the captured images (step S130) and transmits the edited video, i.e., the generated event video, to the user terminal 40 (step S133). This allows the user to obtain their own personalized event video of the event they participated in, enriching their event experience. In particular, it is expected that user satisfaction will be further enhanced by providing the user with video footage showing the user (a specific audience member U) and their favorite performer P together during the event.

[0086] The operation process flow according to this embodiment has been described above. Note that the operation process shown in Figure 5 is just one example, and this disclosure is not limited thereto.

[0087] <<3-2. Control of Movement and Imaging>> Next, we will explain the details of the control of the movement and imaging of the drone 10. Below, as an example, we will explain a case in which the server 20 generates movement path and imaging control information, and the server 20 controls the drone 10 according to the movement path and imaging control information.

[0088] Figure 6 is a flowchart showing the flow of control for the movement and imaging of the drone 10 by the server 20 in this embodiment.

[0089] First, the movement path generation unit 222 of the server 20 determines how to include the subject at each point in the song (subject field of view) based on the song information and subject ratio information (step S203). Figure 7 is a diagram illustrating an example of determining the subject field of view according to the subject ratio information according to this embodiment. As shown in Figure 7, the song information 25 includes information on the song structure, such as intro, A section, B section, etc., and the performance time for each section. The subject ratio information 26 includes information on the proportion of imaging time for each subject, as specified by the subscriber, user A. If there are multiple performers P, a proportion may be specified for each performer. For example, the proportion for performer P2 (guitar) is specified as "high," the proportion for user A and performer P1 (vocals) is specified as "medium," and the proportion for performer P3 (bass) is specified as "low." Note that the method of specifying the proportion of imaging time for each subject is not limited to this, and the proportion may be specified as a numerical value.

[0090] The movement path generation unit 222 determines how to include the subject at each point in the song (i.e., the subject field of view 27) based on the song information 25 and the subject ratio information 26. The determination algorithm is not particularly limited. For example, as shown in Figure 7, the movement path generation unit 222 determines the field of view to include performer P1 for the intro, to include performer P2 and user A first, then user A, then performer P2 in sequence for the A section, to include performers P1 to P3 and user A for the B section, to include performer P1 first, then user A, then performer P1 and user A in sequence for the chorus, and to include performer P2 for the guitar solo. The climactic moments of the song (e.g., the chorus) may be an angle of view that captures user A from the front.

[0091] The determined subject field of view includes more detailed specifications of the field of view and composition. For example, a field of view that shows a close-up of user A's face, a field of view that includes performer P2 and user A from a distance, a field of view that includes performer P1's whole body, a field of view that captures performer P3's profile, a composition in which performer P1 is seen through user A, a composition in which user A is seen through performer P2, a field of view that shows a close-up of performer P2's fingers playing an instrument, a composition in which user A is seen at the left edge of the field of view, etc.

[0092] The movement path generation unit 222 determines the subject field of view for all songs performed at the concert. However, if the user's contract specifies which songs to capture, the movement path generation unit 222 may determine the subject field of view only for the specified songs.

[0093] Next, the movement path generation unit 222 determines, based on the camera information of the ground camera 32, whether the field of view of the ground camera 32 and the field of view of the subject overlap (step S206). It is assumed that the timing and field of view at which the ground camera 32 will capture images during the song are predetermined, and therefore, any overlap in field of view can be determined in advance.

[0094] If the field of view of the ground camera 32 and the field of view of the subject overlap (step S206 / Yes), the process returns to step S203, and the movement path generation unit 222 re-determines the field of view of the subject so that the fields of view do not overlap. This is because it is desirable to acquire images with different fields of view to increase the amount of material used for editing event footage, rather than acquiring images with the same field of view. For example, the field of view that overlaps with the field of view of the ground camera 32 may be changed to a close-up of the subject's fingertips (or a close-up of the fingerings if the subject is a musician), or the drone 10 may increase the proportion of the user A field of view. If the ground camera 32 is not used, step S206 is skipped.

[0095] Next, if the subject's field of view does not overlap with the field of view of the ground camera (step S206 / No), the movement path generation unit 222 generates a movement path and imaging control information based on the subject's position information and the determined subject's field of view (step S209).

[0096] Specifically, the movement path generation unit 222 generates a movement path and imaging control information (imaging direction and field of view (zoom, etc.)) that passes through points where imaging is possible at a determined field of view of the subject, based on the position information of the subject. The movement path generation unit 222 can generate a movement path and imaging control information for each song. The movement path may include a correspondence with the song structure (for example, point 1 in the chorus, point 2 in the A section, etc.). Alternatively, the movement path may include a correspondence with the elapsed time from the start of the song (playing time). For example, the generated movement path may include information such as: from ○ seconds to ○ seconds in the playing time, move from point 1 to point 2 from ○ seconds to ○ seconds, move from point 2 to point 3 from ○ seconds to ○ seconds.

[0097] For example, the movement path generation unit 222 may set the imaging point (i.e., the target point in the movement: WP) in a field of view that includes both user A and performer P to a point that images both from the side if their positions are close, or to a point that images from behind user A if they are far apart. The movement path generation unit 222 may also set the imaging point of the drone 10 so that it does not overlap with the line of sight of user A looking at performer P.

[0098] Next, the movement path generation unit 222 controls the movement and imaging of the drone 10 (step S212). Specifically, the movement path generation unit 222 may transmit the generated movement path and imaging control information to the drone 10 and instruct the drone 10 to perform control according to these. Alternatively, the movement path generation unit 222 may acquire position and attitude information from the drone 10 in real time, recognize the current state of the drone 10, and continuously perform movement control to the next point and imaging control according to the movement path.

[0099] Then, the server 20 acquires the captured images taken by the drone 10 and stores them in the storage unit 230 (step S215).

[0100] The processes described above are not limited to the server 20, but may also be performed by the drone 10, the remote controller 15, or the operator terminal 30.

[0101] (Regarding modification of movement path and imaging control information) The server 20 may modify the movement path and imaging control information as appropriate while controlling the drone 10.

[0102] Figure 8 is a flowchart illustrating the modification process for the movement path and imaging control information according to this embodiment.

[0103] As shown in Figure 8, the server 20 analyzes the images captured by the drone 10 in real time, recognizes the subject's face, and obtains the subject's latest location information (step S223). While the subject's location information can be obtained in advance from seat numbers or standing positions, the latest location information may be actually confirmed to ensure more reliable imaging of the subject. It is also possible that spectators may be in areas where their positions are not fixed, such as standing areas.

[0104] Next, the server 20 determines whether a predetermined subject is included within the field of view at a predetermined time (i.e., according to the determined subject field of view) (step S226).

[0105] If the subject is not included (i.e., the predetermined subject is lost) (step S226 / No), the server 20 modifies the movement path and imaging control (step S229). Specifically, the server 20 may cause the drone 10 to search for the predetermined subject.

[0106] Next, the server 20 determines whether the timing of the live sound being played at the concert venue is correct (step S231). Specifically, the server 20 compares the characteristic points of the live sound with the characteristic points of the recorded sound source included in the song information to determine whether the timing within the song is correct. In the case of live performance, the timing (i.e., the timing of the A section, B section, or chorus, etc.) may be ahead or behind.

[0107] Next, if the timing is incorrect (step S231 / No), the server 20 sets the current timing (step S234). For example, if, based on the elapsed time from the start of the song performance at the concert venue, the timing of the recorded sound source is currently the timing of the B section, but the live sound is delayed and it is the A section, the current timing is set to the A section. After timing setting (correction), the operation process proceeds to step S229. That is, the server 20 corrects the movement path and imaging control information based on the correct current timing (step S229). For example, if the drone 10 had moved to a point corresponding to the B section, the control information is corrected to either immediately return to the point corresponding to the A section or wait until it is the timing of the B section. The correction is reflected in the control of the drone 10.

[0108] The correction process described above may be performed continuously and repeatedly while the drone 10 is being controlled.

[0109] Furthermore, the above modification process is not limited to the server 20, but may also be performed on the drone 10, the remote controller 15, or the operator terminal 30.

[0110] (If there are multiple drones 10) There may be multiple drones 10. Each drone 10 may have a separate contract user registered.

[0111] Server 20 generates movement path and imaging control information for each of the multiple drones 10. At this time, Server 20 may determine whether there is overlap in the field of view at the same time, and if so, it may make the fields of view different. This is to increase the amount of material used for event footage.

[0112] Figure 9 shows a portion of the movement paths of multiple drones according to this embodiment. The left side of Figure 9 shows the movement path of drone 10a contracted by a specific audience member Ua during the A section, along with the imaging direction and field of view at each point. The right side of Figure 9 shows the movement path of drone 10b contracted by a specific audience member Ub during the same A section, along with the imaging direction and field of view at each point. Ground cameras 32a and 32b are also installed at the event venue. Performers P1 to P3 are on the stage.

[0113] Figure 10 shows the superimposed movement paths of drone 10a and drone 10b during the A section of the song according to this embodiment. In the example shown in Figure 10, the fields of view at the same timing during the A section, specifically at the imaging point PW12a of drone 10a and the imaging point PW12b of drone 10b, are almost identical. In this case, the server 20 may shift the imaging timing or change the degree of zoom. The server 20 may also change the field of view to capture the subject from different imaging directions. For example, at imaging point PW12a, the server 20 may set the field of view so that drone 10a captures performer P1 and audience member Ua from the right side of audience member Ua, and at imaging point PW12b, drone 10b captures performer P1 and audience member Ub from the left side of audience member Ub.

[0114] Furthermore, if drones 10a and 10b are imaging performer P at the same time and with the same field of view, server 20 may configure one drone to image the front of performer P and the other to image the back of performer P.

[0115] Furthermore, the same user (audience member U) may contract for multiple drones 10. In this case, the server 20 will set the travel path of the second contracted drone 10 to be different from that of the first contracted drone 10. The number of drones 10 that can be contracted is not limited to two; it may be three or more. Also, different users may contract for a single drone for each song.

[0116] <<3-3. Video Editing Process>> Figure 11 is a flowchart showing an example of the video editing process according to this embodiment.

[0117] As shown in Figure 11, first, the video editing unit 224 of the server 20 acquires each captured image taken by the drone 10 and the ground camera 32 (step S303).

[0118] Next, the video editing unit 224 acquires subject information for each time period in the captured image (step S306). For example, the video editing unit 224 performs face recognition on the captured image and adds information to the captured image indicating which subject is in which time (scene).

[0119] Next, the video editing unit 224 acquires subject ratio information for video editing (step S309). This subject ratio information may be subject ratio information used during image capture, or subject ratio information for video editing.

[0120] Next, the video editing unit 224 determines whether the user who made the editing request is a subscriber or not (step S312).

[0121] If the user is a contractor (step S312 / Yes), the video editing unit 224 obtains the history of the movement path and imaging control information of the contracted drone 10 (step S315).

[0122] Next, the video editing unit 224 acquires the history of the imaging control information of the ground camera 32 (step S318).

[0123] Next, the video editing unit 224 refers to the acquired information and decides which camera's footage (drone camera 10, ground camera 32) to use at each time point (step S321). The video editing unit 224 may prepare multiple patterns of footage to use. If there are different angles of view at the same time, the number of event videos to be generated can be increased.

[0124] Then, the video editing unit 224 performs the generation of event footage (step S324). The video editing unit 224 may generate multiple event videos.

[0125] In this embodiment, it is also possible to generate event videos using images captured by the drone 10 for non-contract users who do not have a contract.

[0126] Specifically, if the user requesting the edit is not a subscriber (step S312 / No), the video editing unit 224 obtains the seat information and facial image of the non-subscriber user (step S327).

[0127] Next, the video editing unit 224 analyzes each image captured by the drone 10 and the ground camera 32 to obtain information about the video in which non-contracted users are shown (step S330). Specifically, this information includes the duration of the video in which non-contracted users are shown, the field of view of the video in which non-contracted users are shown, the composition, etc. If there is video in which non-contracted users are shown, this information can be used to generate event videos for non-contracted users.

[0128] Then, the video editing unit 224 decides which camera's footage (drone camera 10, ground camera 32) to use at each time (step S321), and generates event footage for non-contract users (step S324).

[0129] Furthermore, the server 20 may use not only the images captured by the drone 10 contracted by the user, but also images captured by other drones 10, such as drones 10 contracted by other users, to generate event videos for the user.

[0130] <4. Others> <<4-1. Estimation of the location of a specific spectator U and estimation of related parties>> Figure 12 is a flowchart showing other imaging control processes of the information processing system 1 according to this embodiment.

[0131] As shown in Figure 12, first, the server 20 acquires information to estimate the location of a specific audience member U at the event venue (step S403). The information to estimate the location of a specific audience member U is an example of unique information.

[0132] Next, the server 20 controls the drone 10 at the event venue based on information for estimating the location of a specific audience member U, and performs location estimation processing for that specific audience member U (step S406). Specifically, the server 20 may move the drone 10 to the vicinity of a seat or area based on seat information or seat area information to estimate the location of a specific audience member U. The server 20 considers a person sitting in a seat indicated by the seat information to be a specific audience member U. The server 20 may also identify a specific audience member U from among the people in the area indicated by the seat area information based on appearance information such as a facial image. Alternatively, the server 20 may estimate the location of a specific audience member U by searching for matching wireless device information from wireless information acquired by the drone 10 patrolling the venue, based on wireless device information such as the tethering AP (access point) name and MAC address (Media Access Control address) of an information processing terminal carried by the audience member U, and mapping it on a map. Furthermore, the server 20 may use the facial image, height, clothing, and other appearance information of a specific spectator U to estimate the location of that spectator U from the images captured by the drone 10 as it patrols the venue.

[0133] Next, if the server 20 finds a specific audience member U (step S409 / Yes), it analyzes the images captured by the drone 10 and estimates the level of intimacy between the specific audience member U and other audience members Ur who are in the vicinity of the specific audience member U (step S412).

[0134] For example, if there is eye contact between a specific audience member U and another audience member Ur, server 20 estimates the other audience member Ur to have a closeness level of 1. Also, if the specific audience member U and the other audience member Ur have a conversation totaling N seconds or less, server 20 estimates the other audience member Ur to have a closeness level of 2. Also, if the specific audience member U and the other audience member Ur have a conversation totaling more than N seconds, server 20 estimates the other audience member Ur to have a closeness level of 3. Also, if there is physical contact between the specific audience member U and the other audience member Ur, server 20 estimates the closeness level of 4.

[0135] Next, the server 20 registers the estimated intimacy level of other audience members Ur (step S418).

[0136] If the specific spectator U is not found in step S409 (step S409 / No), the server 20 repeats the process in steps S406 to S409 until the total search time for the specific spectator U exceeds a predetermined time (step S421 / No). If the total search time exceeds a predetermined time (step S421 / Yes), the server 20 interrupts the search process for the specific spectator U.

[0137] The intimacy levels of other registered audience members Ur may be used, for example, when generating event videos. For privacy reasons, server 20 may process the appearance of other audience members in the captured images used to generate event videos, such as blurring the faces of specific audience members Ur. In this case, server 20 may choose not to process certain other audience members Ur, or may change the degree of processing depending on their intimacy level. For example, server 20 may choose not to blur the faces of other audience members Ur with an intimacy level of 3 or higher.

[0138] Furthermore, the intimacy levels of other registered spectators Ur may be used when controlling the imaging by the drone 10. For example, the server 20 may capture a specific spectator U and another spectator Ur with an intimacy level of 4 in a single field of view, or capture close-ups of both.

[0139] The methods for estimating the location (search) of a specific audience member U and estimating their level of intimacy, as described above, are merely examples, and this embodiment is not limited thereto.

[0140] <<4-2. Other Examples of Image Control>> Next, we will explain other examples of image control using the drone 10.

[0141] (Image capture control according to the position estimation of the drone 10) The server 20 may estimate the position of a specific audience member U within the event venue during the event using position estimation as described with reference to Figure 12, and control the drone 10 to capture images of the specific audience member U. The server 20 may also control the image capture to include the specific audience member U and performer P in the field of view. If there are multiple performers P, and the specific audience member U has specified their favorite performer Pn, the server 20 may control the image capture to capture an angle of view that includes performer Pn and the specific audience member U. If the second favorite performer Pm has been specified, the server 20 may control the image capture to capture an angle of view that includes performer Pm and the specific audience member U, or an angle of view that includes performer Pn and performer Pm, or an angle of view that includes performer Pn, performer Pm, and the specific audience member U.

[0142] Server 20 may determine the performer P's position during the event based on pre-registered standing position information, and may further identify performer P through facial image recognition. Alternatively, Server 20 may determine performer P's position based on wireless device information from wireless devices possessed or worn by performer P.

[0143] Server 20 may move the drone 10 to a position where the profile or front view of the performer P and a specific audience member U is within the field of view and they appear as large as possible (highest pixel count). In this case, machine learning or AI (Artificial Intelligence) may be used to determine whether it is a profile or a front view of the face.

[0144] (Consideration of the number of pixels of the subject) The server 20 may also determine whether the area of ​​the subject's face exceeds a threshold number of pixels and control the imaging position of the drone 10 as appropriate.

[0145] Figure 13 is a diagram illustrating an example of the imaging position of the drone 10a according to this embodiment. As shown in Figure 13, when the performer P and a specific audience member U are in relatively close proximity, it is possible to capture images of both of their faces with a pixel count above a threshold using a single drone 10a.

[0146] On the other hand, if performer P and a specific audience member U are relatively far apart, the drone 10a would need to move away from the subjects in order to capture both performer P and audience member U in the frame, making it difficult to photograph both faces with a pixel count above the threshold using a single drone 10a. Furthermore, even if the camera performance can meet the pixel count threshold by moving away from the subjects, the size of the event venue may prevent the drone 10a from moving back. In such cases, the server 20 may issue a warning.

[0147] Figure 14 is a diagram illustrating how to handle situations where imaging is difficult according to this embodiment. The upper part of Figure 14 shows a case where the performer P and a specific audience member U are in relatively distant locations, and the drone 10a moves away from the subject in order to fit both into the field of view. In this case, if the area of ​​the subject's face within the field of view falls below a threshold number of pixels, the server 20 issues a warning.

[0148] In response to the warning, server 20 or the operator deploys multiple drones (specifically, a second drone 10b) to the event venue.

[0149] The lower part of Figure 14 shows the case where drone 10b is added. As shown in the lower part of Figure 14, the server 20 may position drones 10a and 10b so that the profile or front view of performer P and a specific audience member U can be captured with a number of pixels equal to or greater than a threshold. When generating event video, the server 20 may generate an image (a single image divided into two parts) consisting of the image of performer P captured by drone 10a and the image of the specific audience member U captured by drone 10b, and thus pseudo to fit performer P and the specific audience member U into a single field of view.

[0150] The server 20 may also capture images of either the performer P or a specific audience member U, or both, using a camera other than the drone 10 (a ground camera 32 or another autonomous mobile camera).

[0151] (Drone 10 equipped with a 360° camera or multiple cameras) It is also possible that the camera (imaging unit 120) mounted on the drone 10 is a 360° camera or multiple cameras. In that case, the server 20 may position the drone 10 to the side of the performer P and the specific audience member U, and image the performer P and the specific audience member U together from the side. In particular, in the case of a drone 10 equipped with multiple cameras, each camera can be controlled to its own imaging direction, so even if the performer P and the specific audience member U are far apart, they can each be included in the field of view.

[0152] Furthermore, the server 20 may position the drone 10, which is equipped with a 360° camera or multiple cameras, in front of the performer P and a specific audience member U, and capture images of the performer P and the specific audience member U together from the front. Figure 15 shows an example of the frontal positioning of the drone 10 according to this embodiment.

[0153] Specifically, the upper part of Figure 15 shows an example of the arrangement of a drone 10d equipped with a 360° camera, and the lower part of Figure 15 shows an example of the arrangement of a drone 10e equipped with multiple cameras. When the drones 10d and 10e are positioned between the performer P and a specific audience member U, they are capable of capturing images of the performer P and the specific audience member U from the front.

[0154] (Drone 10 separation control) When imaging performer P and a specific audience member U from the front, the server 20 may perform separation control to position the drone 10 so that it is not directly in front of performer P and the specific audience member U. This is because if the drone 10 is positioned directly in front, or more specifically, in the line of sight of the specific audience member U toward performer P, or in the line of sight of performer P, the view will be obstructed.

[0155] The server 20 may estimate the facial orientation and gaze direction of performer P and specific audience member U by image recognition based on images captured by the drone 10, and further estimate the field of view of the gaze from the appearance of performer P and specific audience member U, and then control the distance of the drone 10.

[0156] For example, the range of what can be clearly perceived in the field of view of both human eyes is said to be 30 to 60°. When fixating on something, the range becomes even narrower. Therefore, the server 20 may control the drone 10 to move away from it based on a cone with a vertex angle of 60° centered on the estimated line of sight. Alternatively, the server 20 may estimate the state of the performer P or a specific audience member U from the images captured by the drone 10, estimate whether they are gazing vaguely or intently, and then narrow the vertex angle of the cone to control the separation.

[0157] Figure 16 is a diagram illustrating the separation control of the drone 10 according to this embodiment. Specifically, the upper part of Figure 16 shows an example of a separation-controlled arrangement of a drone 10d equipped with a 360° camera, and the lower part of Figure 16 shows an example of a separation-controlled arrangement of a drone 10e equipped with multiple cameras.

[0158] In Figure 16, the field of view of both human eyes is defined as the field of view angle V (Vp, Vu), and the positions of the drones 10d and 10e are controlled to move horizontally away from each other to avoid the field of view angle V. Note that the separation control is not limited to horizontal movement, but may also be vertical movement. Furthermore, if there are two or more performers P or specific audience members U, the server 20 may similarly estimate each person's field of view angle and control the separation of the drones 10 accordingly.

[0159] The above describes the case where the drone 10 is positioned in front. In particular, when the performer P and a specific audience member U are far away, and it is difficult to position the drone 10 to the side of the performer P and the specific audience member U (which may result in limitations on the number of pixels and composition of the subject's image), positioning the drone 10 in front maximizes the number of pixels of the subject's face and optimizes the imaging results.

[0160] Detachment control is applicable not only to drones 10 equipped with a 360° camera or multiple cameras, but also to drones 10 equipped with a single standard camera.

[0161] The above describes how the drone 10's distance is controlled by estimating the subject's line of sight, but other methods are also possible. For example, the distance control could be performed according to the weight and dimensions of the drone 10 relative to the distance from the positions of the performer P and a specific audience member U. For example, if the drone 10 is small, it may not be noticeable even if it is in the line of sight, depending on the distance to the performer P or the specific audience member U.

[0162] Alternatively, instead of using a pixel count threshold, the operator or a specific spectator U may pre-determine the position of the drone 10.

[0163] (Movement control of drone 10) After the server 20 has positioned performer P and a specific audience member U within its field of view, the drone 10 may move based on a predetermined path, sound and light information obtained within the venue, commands obtained via wireless communication, or designation of a specific audience member U, thereby changing the field of view and composition of performer P and the specific audience member U.

[0164] For example, if one drone 10 is in operation and the drone 10 is equipped with one camera, it may not be possible to capture images that include both the performer P and a specific audience member U within the frame.

[0165] Figure 17 shows examples of changes in the imaging capabilities of the drone 10a according to this embodiment. The upper part of Figure 17 shows example 1 of the changes, and the lower part of Figure 17 shows example 2 of the changes.

[0166] In the variation example 1 shown in the upper part of Figure 17, the server 20 sets a point located to the side of a specific audience member U and performer P as the initial imaging point (point 1), and first uses the drone 10 to image both the specific audience member U and performer P from the side. Next, the server 20 moves the drone 10 to a point (point 2) where performer P's face can be imaged from the front, and images performer P's face from the front. Then, the server 20 moves the drone 10 to a point (point 3) where the face of the specific audience member U can be imaged from the front (forward and upward, etc.), and images the face of the specific audience member U from the front. Finally, the server 20 may return the drone 10 to the initial imaging point.

[0167] When controlling the drone 10 to move through each point, the server 20 may change the flight time at each point (i.e., the time spent stationary in the air, corresponding to the length of imaging time at that point) according to the conditions of the event venue, the movements of performer P or a specific audience member U, or a percentage (subject ratio information) specified in advance by a specific audience member U. The conditions of the event venue include, for example, the level of excitement based on the images and audio data obtained by the drone 10, or the progress of the event (including the progress of the music) as determined from the event's timetable.

[0168] Furthermore, the server 20 may control the drone 10's camera to zoom in at point 2 or point 3 to capture images of performer P or a specific audience member U. Also, in the upper part of Figure 17, the drone 10's movement path is between performer P and a specific audience member U, but if there is a flyable area behind performer P, the server 20 may fly behind performer P to capture images of the specific audience member U.

[0169] Furthermore, as shown in the lower part of Figure 17, the server 20 may control the position and camera direction (imaging direction) of the drone 10, without changing the fact that the drone 10 is positioned to the side of the performer P and the specific audience member U, thereby changing the field of view that includes the performer P and the specific audience member U from the side. For example, the server 20 may change the position of the drone 10 in an arc shape with respect to the center point between the position of the performer P and the position of the specific audience member U.

[0170] (Imaging that includes the subject displayed on the screen in the field of view) Server 20 may use a ground camera 32 or the like to capture an image of a specific audience member U in the concert venue E and display it in real time on a screen on the stage S, and use a drone 10 to capture an image of the performer P and the specific audience member U together. Alternatively, Server 20 may display a large image of the performer P in real time on a screen on the stage S and use the drone 10 to capture the image of the performer P together with the specific audience member U who has their back to the screen. This makes it possible to obtain an image with sufficient visibility that includes both the performer P and the specific audience member U in the field of view, even if the specific audience member U is in a seat far from the stage S.

[0171] (Note) The imaging control and movement control of the drone 10 described above may be performed not only by the server 20, but also by the drone 10, the remote controller 15, or the operator terminal 30, etc.

[0172] Furthermore, the imaging control and movement control of the drone 10 may be controlled in real time and flexibly according to the subject's information (location information, facial image, etc.) and the conditions of the event venue.

[0173] Furthermore, the method for determining the imaging position (placement of the drone 10) when the performer P and a specific audience member U are close or far apart, the method for positioning the drone 10 when the camera mounted on the drone 10 is a 360° camera or multiple cameras, separation control, and changes in the field of view or composition may be used in advance when the server 20 or the like generates movement path and imaging control information.

[0174] <5. Modifications> In the embodiments described above, a concert was given as an example of an event, but this disclosure is not limited thereto, and for example, it may be a competition. In this case, the event venue corresponds to the competition venue, the performers correspond to the competitors, and the audience watching the event corresponds to the spectators.

[0175] Figure 18 is a diagram illustrating imaging by a drone 10 at competition venue F according to a modified version of this embodiment. As shown in Figure 18, competitors I1 and I2 are competing at competition venue F, and imaging is performed by a drone 10 flying within competition venue F. The system configuration is the same as that shown in Figure 2. The configuration of the drone 10 and the server 20 are also the same as those shown in Figures 3 and 4.

[0176] In the aforementioned concert, the position of performer P is predetermined, and based on the positional information of specific audience members U and performer P, it is possible to pre-determine the imaging points, movement paths, and imaging control information. However, in a competition, the position of competitor I is not constant during the competition. The position estimation of competitor I may be performed in real time based on information manually entered by the operator from the operator terminal 30, the results of analysis of images captured by the drone 10 or ground camera 32, etc., and the detection results of sensors other than cameras (infrared sensors, ultrasonic sensors, etc.) installed at the competition venue F. Alternatively, the position estimation of each competitor I may be performed based on communication with wireless devices possessed or worn by each competitor I.

[0177] Furthermore, in a competition, competitor I may make a sudden movement and move out of the frame. The movement path generation unit 222 of the server 20 predicts the next movement of each competitor I and generates a movement path, so it can respond to sudden changes in circumstances and capture images of the subject with an appropriate frame, for example, a frame that includes the front of competitor I's face.

[0178] <<5-1. Operation Processing>> Figure 19 is a flowchart showing an example of the drone control flow at a competition venue according to a modified version of this embodiment.

[0179] As shown in Figure 19, first, the drone control unit 223 of the server 20 starts imaging each competitor I with the drone 10 at the competition venue F (step S503). Specifically, the server 20 flies the drone 10 to the competition venue F and starts imaging with the camera (imaging unit 120). If there are multiple competition areas within the competition venue F, such as in judo, karate, fencing, wrestling, table tennis, and tennis matches, the server 20 may acquire information on the competition area as location information for the competitors I in advance, move the drone 10 to the competition area, and then start imaging.

[0180] Next, the drone control unit 223 of the server 20 performs imaging control to include each competitor I in the field of view (step S506). The server 20 determines whether each competitor I is included in the field of view by, for example, image recognition (also called face recognition) based on the face image of the competitor I. The server 20 continues face recognition to search for each competitor I from the surroundings. The server 20 may also perform movement and imaging control according to pre-generated movement path and imaging control information (for example, a route that circles around the competitor I and sequentially takes images from the front, side, back, and above).

[0181] Next, the movement path generation unit 222 of the server 20 predicts the movements of each competitor I up to XX seconds later (step S509). Specifically, the movement path generation unit 222 makes predictions based on the competitor information of each competitor I (an example of performer behavioral tendency information). The competitor information includes the tendency of competitor I's movements in the competition (what kind of movements they often make in the competition), their special skills, movements they are not good at, and the characteristics of competitor I's coach or supervisor. Competitor information is an example of subject-specific information. The movement path generation unit 222 predicts the movements of competitor I1 and competitor I2, or competitor I1 or competitor I2, up to XX seconds later, based on the competitor information of competitor I1 and the competitor information of competitor I2. More specifically, the movement path generation unit 222 may predict changes in movement corresponding to the time series from the current time to XX seconds later. If athlete information is unavailable, the movement path generation unit 222 may use template athlete information (information on the average movement in the relevant competition) to make a prediction.

[0182] Next, the movement path generation unit 222 of the server 20 generates new movement path and imaging control information based on the prediction results (step S512). The movement path generation unit 222 may also modify the generated movement path and imaging control information up to XX seconds later. If there is a pre-set priority imaging specification, the movement path generation unit 222 generates new movement path and imaging control information taking the priority imaging specification into consideration. Priority imaging specifications include, for example, the angle of view, composition, or which competitor I should prioritize imaging (subject ratio information). Priority imaging specifications may be specified by the user who has contracted for the drone 10. In addition, the movement path generation unit 222 may generate new movement path and imaging control information according to basic rules such as ensuring that the area of ​​competitor I within the angle of view is Y% or more, regardless of whether or not there is a priority imaging specification.

[0183] Next, the drone control unit 223 of the server 20 controls the drone 10 to follow the generated movement path and imaging control information (step S515).

[0184] Next, the movement path generation unit 222 of the server 20 determines whether the movement of competitor I is as predicted (step S518). The movement path generation unit 222 analyzes the captured images received from the drone 10 and can detect the movement of competitor I in real time. The movement path generation unit 222 also determines whether the movement of competitor I is as predicted every X seconds (X seconds < XX seconds).

[0185] Next, if the movements of competitor I are as predicted (step S518 / Yes), the drone control unit 223 of the server 20 continues to track the movement path and imaging control information based on the prediction results, and reflects the prediction results up to the present in the competitor information (step S521).

[0186] The server 20's movement path generation unit 222 repeats the decision process in step S518 every X seconds, for example, until the predicted XX seconds have elapsed (step S524).

[0187] On the other hand, if competitor I's movements are not as predicted (step S518 / No), the drone control unit 223 of the server 20 stops tracking the movement path and imaging control information based on the prediction results (step S527). If competitor I's movements are not as predicted, it means that competitor I has moved in a way different from what was predicted, and it is also assumed that competitor I has been lost from the field of view. In this case, the drone control unit 223 stops drone control according to the prediction results. The drone control unit 223 searches for competitor I and resumes imaging with competitor I included in the field of view, as shown in step S506.

[0188] The processes shown in steps S506 to S527 above are repeated until the competition ends (step S530).

[0189] The predictions made by competitor I and the drone control based on those predictions, as described above, may be performed not only on the server 20, but also on the drone 10, or on the remote controller 15 or the operator terminal 30.

[0190] Furthermore, in the case of team competitions involving multiple athletes, the athlete information used for prediction may include team information (an example of team behavioral tendency information). Team information includes information such as the team's competitive tendencies. In addition, in the case of team competitions, the overall situation of the team may be included in the prediction. Furthermore, in this embodiment, in the case of team competitions, imaging and prediction may be performed focusing on a specific athlete I based on the team information.

[0191] <<5-2. Image Capture Including Spectator T>> Next, we will explain image capture that includes not only competitor I but also spectator T. In this embodiment, it is possible to accommodate spectator T's request to "acquire event footage that includes myself."

[0192] Based on the location information of a specific spectator T and a competitor I, the server 20 first moves the drone 10 to its initial position and controls the drone 10 to capture images of the specific spectator T and competitor I (I1, I2) at the initial position. Next, the server 20 moves the drone 10 from the initial position and controls it to capture images with a field of view that includes the face of the specific spectator T and the figures of competitors I1 and I2. Subsequently, the server 20 moves the drone 10 to the next location and controls it to capture images of competitors I1 and I2.

[0193] The server 20 may generate a travel path passing through each of the above-mentioned points based on the location information of a specific spectator T and athlete I, subject ratio information, and at least the required field of view (for example, the field of view from directly in front of the face).

[0194] Figure 20 illustrates an example of drone control at the competition venue F using a modified version of this embodiment. The location of a specific spectator T can be determined by seat number, etc. The locations of competitors I1 and I2 can be determined in real time by the location estimation described above.

[0195] As shown in Figure 20, the server 20 controls the drone 10 at the competition venue F and takes images of a specific spectator T and competitors I1 and I2 as appropriate. Specifically, the server 20 controls the drone 10 to take images of the specific spectator T and competitors I1 and I2 together from the side at point 1 shown in Figure 20, then to take images of the front of the face of the specific spectator T and the front of the face of competitor I2 from behind competitor I1 at point 2, and finally to take images of the front of the face of competitor I1 from behind competitor I2 at point 3.

[0196] Furthermore, if the server 20 can capture images of the frontal views of multiple people at a single location, it may reduce the number of locations the drone 10 moves to. Also, the server 20 may increase the number of locations the drone 10 moves to depending on the number of specific spectators T or competitors I.

[0197] Furthermore, the server 20 may move the drone 10 to a position where the profile or front view of a specific spectator T and competitor I is within the field of view and they appear as large (highest pixel count). In this case, machine learning or AI may be used to determine whether or not it is a profile or a front view of the face.

[0198] If the competitors I are far apart, or if a specific spectator T is far away from a competitor I, the drone 10 needs to move away from the subjects in order to capture both competitor I and the specific spectator T in the frame, and it may be difficult to capture both of their faces with a pixel count above the threshold using a single drone 10. In this case, the server 20 may issue a warning. Additional drones 10 will be deployed in response to the warning. The server 20 may position each drone 10 so that the profile or front view of the specific spectator T and competitor I can be captured with a pixel count above the threshold. The server 20 may also capture either or both of the specific spectator T and competitor I using a camera other than the drone 10 (ground camera 32, or other autonomous mobile camera).

[0199] In this modified version, the same user (spectator T) may contract for multiple drones 10. In this case, the server 20 will set the imaging location and field of view of the second contracted drone 10 to be different from that of the first contracted drone 10. This is to increase the amount of material for generating event footage. The number of drones 10 that can be contracted is not limited to two, but may be three or more. Also, a different user may contract for a single drone for each match.

[0200] Furthermore, the server 20 may, as in the example shown in the lower part of Figure 17, maintain the position of the drone 10 to the side of the spectator T and competitor I, and control the position and camera direction (imaging direction) of the drone 10 to change the field of view that includes the spectator T and competitor I from the side.

[0201] Figures 21 and 22 illustrate another example of drone control at competition venue F according to a modification of this embodiment. As shown in Figure 21, if competitors I1 and I2 move significantly while competing at competition venue F, the drone 10 may change its initially planned movement path (a movement path passing through points 1, 2, and 3) as shown in Figure 21 to the movement path shown in Figure 22.

[0202] As competitors I1 and I2 move, they move out of the field of view at point 1 in Figure 21. Therefore, server 20 moves drone 10 to point 1' in Figure 22 and, based on the latest position information of competitors I1 and I2, determines points 2' and 3' so that competitors I1 and I2 and spectator T can be imaged from the front, and generates a new movement path passing through these points.

[0203] The image capture control, including the spectator T, and the change of the movement path by the drone 10 as described above may be performed not only by the server 20, but also by the drone 10, or by the remote controller 15 or the operator terminal 30.

[0204] <<5-3. Video Editing>> In addition, the server 20 can perform video editing using the images captured by the drone 10, similar to the embodiment described above, to generate event video for a specific spectator T. The details are the same as the video editing process in the embodiment described above with reference to Figure 11, etc. Furthermore, the server 20 can also generate event video using images captured by a camera other than the drone 10, for example, a ground camera 32, similar to the embodiment described above.

[0205] <6. Hardware Configuration> An embodiment of the present disclosure has been described above. Next, with reference to Figure 23, an example of a hardware configuration used in a drone 10, server 20, remote controller 15, operator terminal 30, or user terminal 40 according to an embodiment of the present disclosure will be described.

[0206] Figure 23 is a block diagram showing an example of the hardware configuration of an information processing device 900 according to one embodiment of the present disclosure. The information processing device 900 is an example of a hardware configuration applicable to the drone 10, server 20, remote controller 15, operator terminal 30, or user terminal 40 according to this embodiment. Note that the information processing device 900 does not necessarily have all of the hardware configurations shown in Figure 23.

[0207] As shown in Figure 23, the information processing device 900 includes a processing circuit 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925.

[0208] The processing circuit 901 functions as an arithmetic processing unit and control unit, and controls the overall operation or a part of the operation within the information processing unit 900 according to various programs recorded in the ROM 902, RAM 903, storage device 919, or removable recording medium 927. The ROM 902 stores programs and calculation parameters used by the processing circuit 901. The RAM 903 temporarily stores programs used in the execution of the processing circuit 901 and parameters that change as appropriate during its execution. The processing circuit 901, ROM 902, and RAM 903 are interconnected by a host bus 907, which is composed of an internal bus. Furthermore, the host bus 907 is connected to an external bus 911, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 909.

[0209] The input device 915 is a device operated by the user, such as a button. The input device 915 may also include a mouse, keyboard, touch panel, switch, and lever. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an external connection device 929 such as a mobile phone that is compatible with the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs it to the processing circuit 901. By operating this input device 915, the user inputs various data to the information processing device 900 or instructs it to perform processing operations.

[0210] The input device 915 may also include an imaging device and sensors. The imaging device is a device that captures real space and generates an image using various components such as an image sensor, such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and a lens for controlling the imaging of a subject onto the image sensor. The imaging device may capture still images or motion images. The sensors are various types of sensors, such as distance sensors, acceleration sensors, gyro sensors, geomagnetic sensors, vibration sensors, light sensors, and sound sensors. The sensors acquire information about the state of the information processing device 900 itself, such as the orientation of the housing of the information processing device 900, and information about the surrounding environment of the information processing device 900, such as the brightness and noise around the information processing device 900. The sensors may also include a GPS sensor that receives GPS (Global Positioning System) signals and measures the latitude, longitude, and altitude of the device.

[0211] The output device 917 is comprised of a device capable of visually or audibly notifying the user of the acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker or headphones. The output device 917 may also include a PDP (Plasma Display Panel), a projector, a hologram, a printer, etc. The output device 917 outputs the results obtained from the processing of the information processing device 900 as text or images, or as sound such as voice or acoustics. The output device 917 may also include a lighting device that brightens the surroundings.

[0212] The storage device 919 is a data storage device configured as an example of the storage unit of the information processing device 900. The storage device 919 is composed of, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. This storage device 919 stores programs and various data executed by the processing circuit 901, as well as various data acquired from external sources.

[0213] The drive 921 is a reader / writer for removable recording media 927 such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory, and is either built into or external to the information processing device 900. The drive 921 reads information recorded on the installed removable recording media 927 and outputs it to the RAM 905. The drive 921 also writes data to the installed removable recording media 927.

[0214] The connection port 923 is a port for directly connecting equipment to the information processing device 900. The connection port 923 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. Alternatively, the connection port 923 may be an RS-232C port, an optical audio terminal, or an HDMI (High-Definition Multimedia Interface) port. By connecting an external device 929 to the connection port 923, various types of data can be exchanged between the information processing device 900 and the external device 929.

[0215] The communication device 925 is a communication interface, for example, consisting of a communication device for connecting to an external network 60. The communication device 925 may be, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth®, Wi-Fi®, or WUSB (Wireless USB). Alternatively, the communication device 925 may be a router for optical communication, an ADSL (Asymmetric Digital Subscriber Line) router, or a modem for various types of communication. The communication device 925 transmits and receives signals, for example, to the Internet or other communication devices using a predetermined protocol such as TCP / IP. The external network 60 connected to the communication device 925 is a network connected by wire or wireless, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0216] For example, when the information processing device 900 functions as a drone 10 or server 20 according to the embodiment of this disclosure, the processing circuit 901 of the information processing device 900 functions as a control unit 130 or control unit 220 by executing a program loaded on the RAM 903. The storage device 919 stores the information processing program according to this disclosure and various data stored in the storage unit 150 or storage unit 230.

[0217] The processing circuit 901 reads and executes program data from the storage device 919, but as an alternative, these programs may be obtained from other devices via the external network 50. In other words, the storage device 919 is not limited to being inside the information processing device 900, but may be located outside the information processing device 900.

[0218] The processing circuit 901 is an example of an integrated circuit, and CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), ASIC (Application Specific Integrated Circuit), and FPGA (Field Programmable Gate Array) can all be considered integrated circuits.

[0219] Furthermore, when the information processing device 900 functions as a drone 10 or server 20 according to the embodiment of this disclosure, the communication device 925 corresponds to the communication unit 110 or the communication unit 210. The output device 917 may also be a mechanical output device and corresponds to the drive unit 160. The input device 915 corresponds to the imaging unit 120 or the sensor unit 140.

[0220] <7. Supplementary Information> Although preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the present technology is not limited to such examples. It is clear that a person with ordinary skill in the art of the present disclosure may conceive of various modifications or alterations within the scope of the technical idea described in the claims, and these will naturally be understood to fall within the technical scope of the present disclosure.

[0221] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0222] The information processing system according to this disclosure may correspond to a single device consisting of a drone 10, a server 20, a remote controller 15, an operator terminal 30, or a user terminal 40. Alternatively, the information processing system according to this disclosure may consist of multiple devices. A combination of multiple devices may be, for example, a drone 10, a server 20, a remote controller 15, an operator terminal 30, or a user terminal 40.

[0223] Furthermore, the embodiments and modifications of this disclosure described above can be combined as appropriate in areas where the processing content is not contradictory. Also, the order of each step shown in the sequence diagram or flowchart of this embodiment can be changed as appropriate. For example, each step may be processed chronologically, repeatedly, or partially in parallel.

[0224] Furthermore, it is possible to create one or more computer programs for the CPU, ROM, RAM, and other hardware built into the drone 10, server 20, remote controller 15, operator terminal 30, or user terminal 40 to perform the functions of the drone 10, server 20, remote controller 15, operator terminal 30, or user terminal 40. A computer-readable storage medium storing such one or more computer programs is also provided.

[0225] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein.

[0226] Furthermore, this technology can also be configured as follows: (1) An information processing method comprising: a processor controlling the imaging of one or more movable mobile imaging devices that image one or more subjects; generating imaging control information and a movement path that enables imaging in which one or more subjects are included in the field of view, based on specific information relating to the subjects, and controlling the movement of one or more of the mobile imaging devices. (2) The information processing method according to (1), wherein one or more subjects are at least one of spectators watching an event and performers performing at the event. (3) The information processing method according to (2), wherein the processor performs imaging in which a specific spectator and performer are included in the field of view. (4) The information processing method according to (2) or (3), wherein the performance includes at least one of a show, acting, singing, playing an instrument, and a competition, and the performers include at least one of performers, musicians, and competitors. (5) The information processing method according to any one of (2) to (4), wherein the processor controls the mobile imaging device to continuously capture images during the event, and the video captured by the mobile imaging device includes at least scenes that include specific audience members and performers in the field of view. (6) The information processing method according to any one of (2) to (5), wherein the specific information relating to the subject includes subject ratio information indicating the temporal proportion of capturing specific audience members and performers during continuous imaging during the event, and the processor generates the imaging control information and movement path during the event based on the subject ratio information. (7) The information processing method according to (6), wherein the subject ratio information further includes composition information. (8) The information processing method according to (6) or (7), wherein the event is a concert, and the processor generates, based on song information indicating the performance time and structure of songs to be performed at the concert, and subject ratio information, the movement path of the mobile imaging device while the songs are being performed, and imaging control information including information on the imaging direction and field of view along the movement path.(9) The information processing method according to any one of (2) to (8), wherein the specific information relating to the subject includes location information of a specific audience member and a performer within the event venue, and the processor generates the imaging control information and a movement path based on the location information. (10) The information processing method according to (9), wherein the location information of the specific audience member is three-dimensional location information based on the seat number of the specific audience member. (11) The information processing method according to (9), wherein the location information of the specific audience member is three-dimensional location information obtained by searching for an information processing terminal held by the specific audience member within the event venue, or by searching by analyzing an image captured based on the face image of the specific audience member within the event venue. (12) The information processing method according to any one of (2) to (11), further comprising: the processor analyzing an image captured by a mobile imaging device controlled by the generated imaging control information and movement path to perform face recognition of the subject; and correcting the imaging control information and movement path if one or more predetermined subjects are not included in the field of view at a predetermined timing. (13) The information processing method according to any one of (2) to (7), wherein the specific information relating to the subject includes behavioral tendency information of a performer or behavioral tendency information of a team consisting of multiple performers, and the processor further comprises predicting the performer's behavior based on the behavioral tendency information, and generating the movement path and the imaging control information according to the prediction result. (14) The information processing method according to (13), wherein the event is a competition, and the processor further comprises predicting the next action of the competitor, who is the performer.(15) The information processing method according to (14), further comprising: the processor analyzing captured images captured by the mobile imaging device in real time to continuously detect the actual actions of the athlete; comparing the detected actual actions with a series of behavioral changes included in the prediction results; continuing to control the mobile imaging device according to the prediction results as long as the actual actions are as predicted; and stopping the control of the mobile imaging device according to the prediction results if the actual actions differ from the prediction. (16) The information processing method according to (15), further comprising: the processor controlling the mobile imaging device to include the athlete in the field of view after stopping the control of the mobile imaging device according to the prediction results; and predicting the athlete's next actions again. (17) The information processing method according to any one of (2) to (16), further comprising: the processor editing video or still images captured by one or more mobile imaging devices to generate event video. (18) The information processing method according to (17), wherein the processor extracts scenes in which a specific subject is visible from a plurality of captured images taken during an event, and generates event video that includes at least scenes in which the specific subject is visible. (19) An information processing device comprising a control unit that performs imaging control for one or more movable mobile imaging devices that capture one or more subjects, wherein the control unit further generates imaging control information and movement paths that enable imaging in which one or more subjects are included in the field of view, based on specific information relating to the subjects, and performs movement control for one or more of the mobile imaging devices. (20) A program that causes a computer to function as a control unit that performs imaging control for one or more movable mobile imaging devices that capture one or more subjects, wherein the control unit further generates imaging control information and movement paths that enable imaging in which one or more subjects are included in the field of view, based on specific information relating to the subjects, and performs movement control for one or more of the mobile imaging devices.

[0227] 1 Information Processing System 10 Drone 110 Communication Unit 120 Imaging Unit 130 Control Unit 131 Movement Path Generation Unit 132 Movement Control Unit 133 Imaging Control Unit 134 Position and Attitude Estimation Unit 135 Transmission Control Unit 140 Sensor Unit 150 Storage Unit 160 Drive Unit 15 Remote Controller 20 Server 210 Communication Unit 220 Control Unit 221 Registration Processing Unit 222 Movement Path Generation Unit 223 Drone Control Unit 224 Video Editing Unit 230 Storage Unit 30 Operator Terminal 32 Ground Camera 34 Sound System 40 User Terminal

Claims

1. An information processing method comprising: a processor performing imaging control of one or more movable mobile imaging devices that image one or more subjects; and generating imaging control information and a movement path that enables imaging in which one or more subjects are included in the field of view, based on specific information relating to the subjects, and performing movement control of the one or more mobile imaging devices.

2. The information processing method according to claim 1, wherein one or more subjects are at least one of spectators watching the event and performers performing at the event.

3. The information processing method according to claim 2, wherein the processor performs imaging that includes a specific audience member and the performer in the field of view.

4. The information processing method according to claim 2, wherein the performance includes at least one of a show, acting, singing, playing an instrument, and competing, and the performer includes at least one of a performer, an instrumentalist, and a competitor.

5. The information processing method according to claim 2, wherein the processor controls the mobile imaging device to continuously capture images during the event, and the video captured by the mobile imaging device includes at least scenes that include specific audience members and performers in the field of view.

6. The information processing method according to claim 2, wherein the specific information relating to the subject includes subject ratio information indicating the temporal proportion of imaging of a specific audience member and the performer during continuous imaging during the event, and the processor generates the imaging control information and movement path during the event based on the subject ratio information.

7. The information processing method according to claim 6, wherein the subject ratio information further includes composition information.

8. The information processing method according to claim 6, wherein the event is a concert, and the processor generates, based on song information indicating the performance time and structure of songs to be performed at the concert, and subject ratio information, the movement path of the mobile imaging device while the songs are being performed, and imaging control information including information on the imaging direction and field of view along the movement path.

9. The information processing method according to claim 2, wherein the specific information relating to the subject includes location information of specific audience members and performers within the event venue, and the processor generates the imaging control information and movement path based on the location information.

10. The information processing method according to claim 9, wherein the location information of the specific spectator is three-dimensional location information based on the seat number of the specific spectator.

11. The information processing method according to claim 9, wherein the location information of the specific audience member is three-dimensional location information obtained by searching for an information processing terminal held by the specific audience member within the event venue, or by searching by analyzing captured images based on the facial image of the specific audience member within the event venue.

12. The information processing method according to claim 2, further comprising: the processor analyzing the captured image obtained by the mobile imaging device controlled by the generated imaging control information and movement path to perform face recognition of the subject; and correcting the imaging control information and movement path if one or more predetermined subjects are not included in the field of view at a predetermined timing.

13. The information processing method according to claim 2, wherein the specific information relating to the subject includes behavioral tendency information of a performer or behavioral tendency information of a team consisting of multiple performers, and the processor further includes predicting the behavior of the performer based on the behavioral tendency information, and generating the movement path and the imaging control information according to the prediction result.

14. The information processing method according to claim 13, wherein the event is a competition, and the processor further includes predicting the next action of the competitor, who is the performer.

15. The information processing method according to claim 14, further comprising: the processor analyzing the captured images captured by the mobile imaging device in real time to continuously detect the actual actions of the athlete; comparing the detected actual actions with a series of behavioral changes included in the prediction results; continuing to control the mobile imaging device according to the prediction results as long as the actual actions are as predicted; and stopping the control of the mobile imaging device according to the prediction results if the actual actions differ from the prediction.

16. The information processing method according to claim 15, further comprising the processor stopping control of the mobile imaging device in accordance with the prediction result, controlling the mobile imaging device to include the athlete in the field of view, and then predicting the athlete's next action again.

17. The information processing method according to claim 2, further comprising the processor editing video or still images captured by one or more mobile imaging devices to generate event video.

18. The information processing method according to claim 17, wherein the processor extracts scenes in which a specific subject is shown from a plurality of captured images taken during the event, and generates the event video which includes at least the scenes in which the specific subject is shown.

19. An information processing device comprising a control unit that controls the imaging of one or more movable mobile imaging devices that image one or more subjects, wherein the control unit further generates imaging control information and a movement path that enables imaging in which one or more subjects are included in the field of view, based on specific information relating to the subjects, and controls the movement of the one or more mobile imaging devices.

20. A program that causes a computer to function as a control unit for controlling the imaging of one or more mobile imaging devices that are movable and capture images of one or more subjects, wherein the control unit further generates imaging control information and movement paths that enable imaging in which one or more subjects are included in the field of view, based on specific information relating to the subjects, and controls the movement of one or more of the mobile imaging devices.

Images