Remote control method for terminal

Abstract

A first terminal 10, which communicates with a second terminal 20 for displaying a mouse pointer at a position on a display screen on the basis of an input position change amount of an operator and displays the display screen of the second terminal 20 on a host terminal, transmits first operation information for moving the pointer from an initial position of the pointer on the display screen of a counterpart terminal to a position on the screen of the counterpart terminal corresponding to a position on the display screen of the host terminal designated by a user to the second terminal 20. In addition, the first terminal 10 transmits, to the second terminal 20, second operation information for moving the pointer from a position on the display screen of the host terminal at which a designation by the user was cancelled to the initial position.

Description

Remote control method for terminal

[0001] The present invention relates to remote control of a terminal.

[0002] There is a device that has a mirroring function for displaying the screen displayed on a terminal (hereinafter referred to as the target terminal) to be remotely operated on the display screen of the own terminal, and a function for remotely operating the target terminal by transmitting position information indicating a position specified on the display screen of the own terminal to the target terminal when the user touches the touch panel or the like (see Patent Document 1, etc.).

[0003] Japanese Patent Application Laid-Open No. 2023-177072

[0004] Depending on the type of the target terminal, due to specifications such as the OS (Operating System) of the target terminal, the information that can be received as information representing the operation content of the user may be limited to relative position information (in other words, the movement amount of the operator). That is, when remotely operating this type of target terminal, the only option is to connect a mouse to the terminal as well. That is, even if the own terminal has an input device that outputs an absolute position such as a touch panel that is widely used in mobile terminals or the like, it is virtually impossible to remotely operate the target terminal using this input device, which is inconvenient.

[0005] An object of the present invention is to enable remote operation of a target terminal that accepts only relative position information using an external input device that outputs an absolute position.

[0006] A program according to a first aspect of the present invention connects to a partner terminal that displays an operation object at a position on a display screen based on the amount of position change of an input operator, and causes the computer of a terminal that displays image data received from the partner terminal on its own terminal's display screen to execute: a first acquisition step of acquiring first coordinate information indicating a position on the display screen of the own terminal, which is specified by the user of the own terminal; a first generation step of generating first operation information for moving the operation object from an initial position on the display screen of the partner terminal to a position on the screen of the partner terminal corresponding to the specified position; a first transmission step of transmitting the first operation information to the partner terminal; a second acquisition step of acquiring second coordinate information indicating a position on the display screen of the own terminal, which has been released from the user's specification; a second generation step of generating second operation information for moving the operation object to the initial position; and a second transmission step of transmitting the second operation information to the partner terminal. According to this embodiment, in a device that has the function of performing remote control by mirroring the display screen of the other terminal and transmitting touch position information of the own terminal to the other terminal, even if the other terminal does not accept absolute coordinates, touch operations that specify absolute positions (in addition to other drag operations, etc.) can be reflected in the movement of the pointer.

[0007] In a preferred embodiment, the program further causes the computer to perform a determination step to determine a reference position coordinate on the display screen of the terminal corresponding to the initial position, and in the first generation step, generates the first operation information based on the relative coordinates of the first coordinate information with respect to the reference position coordinate. According to this embodiment, the first operation information can be generated based on the relative coordinates of the reference position coordinate and the first coordinate information.

[0008] In a preferred embodiment, the program causes the computer to further perform a determination step to determine the orientation of the display screen of the other terminal, and in the determination step, the reference position coordinates are determined based on the orientation determined in the determination step. According to this embodiment, even if the orientation of at least one of the screens of the local terminal and the other terminal changes, the correspondence between the screen positions of both is maintained.

[0009] In a preferred embodiment, the program further causes the computer to perform the step of setting a partitioned screen area on the display screen of the local terminal that corresponds to the display screen area of ​​the other terminal when the local terminal is connected simultaneously with a plurality of other terminals, including the other terminal, and in the first generation step, a reference position within the partitioned screen area that corresponds to the initial position is determined.

[0010] In a preferred embodiment, the terminal is connected to a second partner terminal that displays an operation object at a position on a display screen based on the absolute position of an input operator, and is capable of displaying image data received from the second partner terminal on the display screen of the terminal itself. The program further causes the computer to display a predetermined object at the position on the display screen of the terminal itself indicated by the first coordinate information. The predetermined object is displayed in a first manner if the position on the display screen of the terminal itself indicated by the first coordinate information is within a display area corresponding to the partner terminal, and is displayed in a second manner different from the first manner if the position on the display screen of the terminal itself indicated by the first coordinate information is within a display area corresponding to the second partner terminal.

[0011] In another aspect, the present invention provides a terminal having means for connecting to a partner terminal that displays an operation object at a position on a display screen based on the amount of position change of an input operator; means for displaying image data received from the partner terminal on the display screen of the own terminal; first acquisition means for acquiring first coordinate information indicating a position on the display screen of the own terminal as specified by the user; first generation means for generating first operation information for moving the operation object from an initial position on the display screen of the partner terminal to a position on the screen of the partner terminal corresponding to the specified position, based on the first coordinate information; first transmission means for transmitting the first operation information to the partner terminal; second acquisition means for acquiring second coordinate information indicating a position on the display screen of the own terminal after the user's specification has been released; second generation means for generating second operation information for moving the operation object to the initial position; and second transmission means for transmitting the second operation information to the partner terminal.

[0012] In yet another aspect, the present invention provides a computer system comprising: a terminal and a partner terminal that displays an operation object at a position on its display screen based on the amount of position change of an input operator. The terminal includes means for displaying image data received from the partner terminal on its own display screen; first acquisition means for acquiring first coordinate information indicating a position on the display screen of the terminal specified by the user; first generation means for generating first operation information for moving the operation object from an initial position on the display screen of the partner terminal to a position on the screen of the partner terminal corresponding to the specified position, based on the first coordinate information; first transmission means for transmitting the first operation information to the partner terminal; second acquisition means for acquiring second coordinate information indicating a position on the display screen of the terminal after the user's designation has been released; second generation means for generating second operation information for moving the operation object to the initial position; and second transmission means for transmitting the second operation information to the partner terminal.

[0013] (A) and (B) are diagrams illustrating an example configuration of the computer system 1 according to an embodiment of the present invention. A block diagram illustrating an example of the functional configuration of the second terminal 20. A block diagram illustrating an example of the functional configuration of the first terminal 10. A diagram illustrating the correspondence between the position of the first terminal 10 on the display screen and the position of the second terminal 20 on the display screen. A diagram illustrating the correspondence between the position of the first terminal 10 on the display screen and the position of the second terminal 20 on the display screen. A diagram illustrating the correspondence between the position of the first terminal 10 on the display screen and the position of the second terminal 20 on the display screen. A diagram illustrating the correspondence between the position of the first terminal 10 on the display screen and the position of the second terminal 20 on the display screen. A diagram illustrating the generation process of first operation information during non-continuous touch operations. A diagram illustrating the generation process of first operation information during continuous touch operations. A diagram illustrating the generation process of second operation information. A diagram illustrating an example of operation in the first terminal 10. A diagram illustrating the movement of the pointer MP in the second terminal 20 during a tap operation. A diagram illustrating the processing flow in the first terminal 10 and the second terminal 20 during a drag operation. This figure illustrates the movement of pointer MP on the second terminal 20 during a drag operation. This figure illustrates the movement of pointers C1 and C2 on the second terminal 20 in another embodiment of the present invention.

[0014] The embodiments described below are subject to various technically preferred limitations. However, the embodiments of the present invention are not limited to those described below.

[0015] Figures 1(A) and 1(B) show an example configuration of a computer system 1 according to one embodiment of the present invention. The computer system 1 includes a first terminal 10 and a second terminal 20, as shown in Figure 1(A), for example. The first terminal 10 and the second terminal 20 communicate with each other according to Wi-Fi, Bluetooth, or other wireless communication standards.

[0016] The first terminal 10 and the second terminal 20 are information processing terminals such as smartphones, tablet devices, notebook PCs, and desktop PCs. The first terminal 10 has a function to remotely control the second terminal and a function to reproduce (mirror) the contents of the screen displayed on the second terminal onto the screen of the first terminal 10 in real time. Specifically, operation information generated based on operations entered in the first terminal 10 is transmitted from the first terminal 10 to the second terminal, and the second terminal 20 performs image processing and other controls that reflect these operation contents. On the other hand, image data representing the screen displayed on the second terminal 20 is transmitted to the first terminal 10 in real time, and the first terminal 10 performs display processing based on this image data. In addition to image data, audio data may also be transmitted from the second terminal 20 to the first terminal 10, and the first terminal 10 may perform sound output processing (sound synchronization) based on this audio data.

[0017] Furthermore, as shown in Figure 1(B), in the computer system 1, multiple second terminals 20 may be simultaneously connected to a single first terminal 10 by wireless or wired means. That is, the first terminal 10 may be used to remotely control multiple second terminals 20 (second terminal 20-1, second terminal 20-2, second terminal 20-3, second terminal 20-4), and the screens displayed on each second terminal 20 may be reproduced on the screen of the first terminal 10 in real time. For the sake of explanation, unless otherwise specified, it will be assumed below that the one-to-one direct connection configuration shown in Figure 1(A) is adopted. In addition, it will be assumed that at least the first terminal 10 is equipped with a touch panel and has a function to detect the touch position by touching the screen with a finger.

[0018] Figure 2A shows the functions of the second terminal 20. Here, only the functions specific to when the second terminal 20 is connected and receives remote control from the first terminal 10 will be described, and the various functions that are exhibited when the second terminal 20 is used independently will not be explained.

[0019] The second terminal 20 includes a communication unit 210, a display control unit 220, a display unit 230, and a posture sensor 250. The communication unit 210 consists of a communication module and the like, and connects to the first terminal 10 by wire or wireless connection to send and receive various information.

[0020] The display control unit 220 outputs image data representing the image displayed on the display unit 230 in real time and transmits it to the first terminal 10 via the communication unit 210. The attitude sensor 250 consists of a 3-axis accelerometer and a gyroscope, and detects information indicating the attitude of the second terminal 20 (for example, whether a rectangular screen is being used in portrait or landscape orientation), and transmits it to the first terminal 10 via the communication unit 210 as needed. Note that the generation of information indicating the attitude of the second terminal 20 and its transmission to the first terminal 10 may be omitted. This is because, if image data is displayed across the entire display screen of the second terminal, the first terminal 10 can determine the orientation of the second terminal based on the received image data. When the display control unit 220 receives operation information reflecting the user's operations input at the first terminal 10 via the communication unit 210, it draws an operation object (called a mouse cursor, pointer, etc.; hereinafter simply referred to as "pointer") at the position on the screen corresponding to this operation information.

[0021] Here, the information that the communication unit 210 can receive as information representing the user's operation is limited to relative position information (i.e., information representing the amount of movement (more precisely, the direction and amount of movement) of the operator provided on the second terminal 20 for receiving user operations). Specifically, while it is possible to connect and use a mouse, which is an operation device that outputs the amount of movement (relative position), to the second terminal 20, it is practically impossible to connect and use an operation device that outputs the coordinates of a single point in a two-dimensional area, such as a stylus or tablet device. For this reason, as will be described in detail below, in this embodiment, the first terminal 10 connected to the second terminal 20 generates operation information that the second terminal 20 can handle based on information indicating absolute position and supplies it to the second terminal 20.

[0022] As mentioned above, the second terminal 20 cannot process absolute position information supplied from other devices, but it may have an input device such as a touch panel, like a typical smartphone, and have a function to process absolute position information supplied internally.

[0023] Figure 2B shows an example configuration of the first terminal 10. As shown in the figure, the first terminal 10 comprises a communication unit 110, an input / output unit 120, a storage unit 130, an attitude detection unit 160, and a processing unit 140.

[0024] The communication unit 110 includes a communication module and connection interface, and communicates with the second terminal 20 via a wired or wireless connection, either through a communication network or directly. Specifically, the communication unit 110 supplies image data received from the second terminal 20 to the display control means 141 and supplies information regarding the attitude of the second terminal 20 to the determination means 143. It also transmits operation information generated by the processing unit 140 to the second terminal 20.

[0025] The input / output unit 120 includes a display device (for example, a liquid crystal display) that displays an image represented by image data provided by the processing unit 140, and a transparent touch sensor provided to cover the display surface of the display device. In this embodiment, the display device and the touch sensor form a touch pad. The touch pad outputs operation data indicating the user's touch position on the touch sensor to the processing unit 140. This supplies the processing unit 140 with information indicating the content of the user's operation (specification of a position on the screen).

[0026] In this embodiment, the touch position represented by the operation data indicates a position (i.e., absolute position) in a two-dimensional coordinate system with the upper left corner of the display surface as the origin. While the first terminal 10 in this embodiment includes an input / output unit 120 that functions as a touchpad, if the first terminal 10 includes an external device interface for connecting other external devices such as a USB interface, an input / output device such as a touchpad may be connected to the external device interface. In short, it is sufficient that the first terminal 10 provides a two-dimensional area where the user can specify a position.

[0027] The attitude detection unit 160 includes an acceleration sensor, a gyroscope, etc., and outputs information indicating the attitude of the first terminal 10 (for example, whether it is being used vertically or horizontally) to the processing unit 140.

[0028] The memory unit 130 is a memory consisting of memory elements and the like. This memory pre-stores program PR1 for the processing unit 140 to implement the control method of the present invention. The volatile memory is used by the processing unit 140 as a work area when executing program PR1.

[0029] In addition, the memory unit 130 stores a display area mapping information MAP that describes the correspondence between the position on the touch panel screen (in other words, the two-dimensional input field) of the currently connected second terminal 20 and the position on the screen of the second terminal 20. More specifically, the display area mapping information MAP describes the correspondence between the position coordinates on the screen of the first terminal 10, including the reference position (origin of the coordinate system) on the screen of the first terminal 10, and the position coordinates on the screen of the second terminal 20, including the reference position (origin of the coordinate system) on the screen of the second terminal 20. The reference position on the screen of the second terminal 20 may or may not be the same as the position (initial position) to which the pointer to be drawn on the second terminal 20 should return each time a tap operation on the first terminal 10 is completed, as will be described later.

[0030] This display area mapping information MAP is generated by the display control means 141, which will be described later. In a preferred embodiment, the display area mapping information MAP is determined based on the orientation of the first terminal 10 and the second terminal 20 (whether the screen is used in portrait or landscape orientation, or in any other orientation (rotation angle), etc.), the physical size of the screens of the first terminal 10 and the second terminal 20, the resolution of the screens of the first terminal 10 and the second terminal 20, and an allocation algorithm that defines which area of ​​the display screen of the first terminal 10 is allocated to the screen of the second terminal 20. However, if there is virtually no choice regarding the orientation of the screens of the first terminal 10 and the second terminal 20, or if the algorithm defines the screen allocation without depending on the orientation of the screens of the first terminal 10 and / or the second terminal 20, then information about the orientation of the screens of the first terminal 10 and / or the second terminal 20 does not need to be reflected in the display area mapping information MAP. In this case, the above reference position remains constant with respect to the images displayed on the first terminal 10 and the second terminal 20.

[0031] The display area mapping information MAP may be acquired by the first terminal 10 when the second terminal 20 starts connecting with the first terminal 10, or it may be stored in the storage unit 130 in advance, and the first terminal 10 may read the display area mapping information MAP from the storage unit 130 when it detects the connection of the corresponding second terminal 20. Alternatively, it is preferable to update the display area mapping information MAP based on the changes whenever the orientation of the second terminal 20 or the resolution settings on the second terminal 20 are changed during the connection. This ensures that the correspondence between the position on the screen of the first terminal 10 and the position on the screen of the second terminal 20 is correctly maintained, and the user can specify the intended position on the second terminal 20 by viewing the screen of the second terminal 20 displayed on the first terminal 10 and specifying the position on that screen.

[0032] The following explains an example of the correspondence between the position on the screen (input field) of the first terminal 10 and the position on the screen (input field) of the second terminal 20, using Figures 3A to 3D. Hereafter, the position on the screen of the first terminal 10 will be represented in the XY (uppercase) coordinate system, and the position on the screen of the second terminal 20 will be represented in the xy (lowercase) coordinate system. In other words, the display area mapping information MAP represents the correspondence between (X, Y) and (x, y), or the conversion rule from one to the other.

[0033] In the example shown in Figure 3A, the first terminal 10 and the second terminal 20 have the same screen size and resolution, and both are used in portrait orientation. Since the first terminal 10 and the second terminal 20 are connected, the same image displayed on the screen of the second terminal 20 (a map image in this example) is displayed on the entire screen (display area R1) of the first terminal 10. In this example, the position on the screen of the first terminal 10 is represented in a coordinate system with the left corner of the screen as the origin Po(Xo,Yo), and the position on the screen of the second terminal 20 is represented in a coordinate system with the left corner of the screen as the origin po(xo,yo). P1 and p1 are corresponding positions. The correspondence between the two positions can be represented by a function F that converts the XY coordinate system to the xy coordinate system. In this example, the correspondence is F(X,Y) = (x,y), that is, X = x, Y = y. Furthermore, in both the XY coordinate system and the xy coordinate system, the origin of the coordinate system is not limited to the left corner of the display area. In short, it is sufficient that there is a one-to-one correspondence between any position on the screen of the second terminal 20 (or any position on the image displayed on the second terminal 20) and any position within the display area of ​​the first terminal 10 on which the screen of the second terminal 20 is displayed.

[0034] In the example shown in Figure 3B, the first terminal 10 is used horizontally and is allocated as a region R2 of the display screen of the first terminal 10, while region RF is set as a region not subject to mirroring display, and the screen displayed on the second terminal 20 is reproduced only in region 2. If the X'Y' coordinate system in region R2 is defined as the origin Po(Xb,Yb) as shown in the figure, the components of the X'Y' coordinate system can be represented by a linear combination of the components of the xy coordinate system (i.e., a combination of translation of the origin and rotation of the axes or scaling of the axes), and the correspondence between the position in the XY coordinate system and the position in the xy coordinate system in the display region mapping information MAP can be represented by a transformation function G.

[0035] In the example shown in Figure 3C, the first terminal 10 is used in portrait orientation and the second terminal 20 is used in landscape orientation, with region R3 allocated as the region for mirroring display. As shown in the figure, if the X''Y'' coordinate system is defined with the origin Po(Xc, Yc), the position of X'' can be expressed as a linear combination of the XY coordinate system components, and the correspondence with the position in the xy coordinate system can be expressed in the form of a transformation function H.

[0036] In the example shown in Figure 3D, four second terminals 20 are simultaneously connected to one first terminal 10. In this example, the display screen of the first terminal 10 is divided into four sections, and each section displays the same screen as the corresponding second terminal 20. As shown in the figure, the αβ coordinate system defined by limiting Po(Xd, Yd) corresponds to the xy coordinate system, and the position in the αβ coordinate system can be expressed as a linear combination of the coordinate components of the XY coordinate system. The correspondence between the position in the XY coordinate system and the position in the xy coordinate system can be expressed using the transformation function J.

[0037] Returning to Figure 2B, the details of the processing unit 140 will be explained. The processing unit 140 is composed of one or more processors such as CPUs. The processing unit 140 is an example of a computer in the present invention. When the power (not shown) of the first terminal 10 is turned on, the processing unit 140 reads program PR1 from non-volatile memory to volatile memory and starts executing the read program PR1. The processing unit 140, operating according to program PR1, plays the role of the control center of the first terminal 10. More specifically, the processing unit 140, operating according to program PR1, functions as the acquisition means 142, display control means 141, determination means 143, generation means and transmission means 145 shown in Figure 2.

[0038] The display control means 141 displays image data received from the second terminal 20 via the communication unit 110 in a display area within the display screen determined according to a predetermined allocation algorithm. This enables mirroring of the other party's screen, allowing the user of the first terminal 10 to perform operations to remotely control the second terminal 20 (i.e., operations on the display screen of the second terminal 20) via the input / output unit 120 while referring to the mirrored other party's screen.

[0039] In the following, the display control means 141 displays the entire screen of the other party on the second terminal 20 on the display screen of the first terminal 10, as shown in Figures 3A to 3D, regardless of whether the first terminal 10 is held in a vertical or horizontal orientation. However, only a portion of the other party's screen may be displayed on the display screen of the first terminal 10. In that case, the first terminal 10 can only specify a position within that portion of the screen.

[0040] Here, if the screen size of the display unit of the first terminal 10 and the screen size of the display unit of the second terminal 20 are different, a predetermined algorithm can be used to determine which portion of the screen area of ​​the first terminal 10 to use (in other words, which portion of the area to allocate as the area for mirroring display). When the allocation of the allocated portion of the area is changed, or when the orientation of the display screens of one or both of the first terminal 10 and the second terminal 20 changes, this information is sequentially reflected in the display area mapping information MAP, and as a result, the correspondence between the position on the screen of the first terminal 10 and the position on the screen of the second terminal 20 is always maintained.

[0041] The determination means 143 determines the coordinates of the reference position on the display screen of the first terminal 10 (hereinafter referred to as the reference position coordinates) that correspond to the initial position on the other party's screen of the pointer displayed on the second terminal 20, based on the display area mapping information MAP. For example, as shown in Figure 3A, if the orientation of the other party's screen is vertical and the orientation of the first terminal 10 is vertical, the determination means 143 determines the position Po of the upper left corner in the display area of ​​the display unit of the first terminal 10 as the reference position corresponding to the initial position po on the other party's screen. As shown in Figure 3B, if the orientation of the other party's screen is vertical and the orientation of the first terminal 10 is horizontal, the determination means 143 determines the position of the upper left corner of the other party's screen displayed in the display area of ​​the display unit of the first terminal 10 as the reference position.

[0042] The acquisition means 142 acquires from the input / output unit 120 first coordinate information indicating a position on the display screen (hereinafter referred to as the specified position) that was specified by an operation performed on the input / output unit 120 by the user of the first terminal 10, in order to remotely control the second terminal 20. In addition, the processing unit 140 acquires second coordinate information indicating a position on the display screen of its own terminal after the user's specification has been released. Specifically, it detects that the position specification has been released (for example, when a finger that was touching the screen leaves the screen). If the specified position indicated by the first coordinate information and the second coordinate information are the same, it corresponds to the position of the touch operation on the display screen of the second terminal 20 if the user remotely controlling the second terminal 20 were to directly operate the second terminal 20.

[0043] Note that the processing unit 140 can also accept a position designation in a mode where the user of the first terminal 10 continuously moves a finger or an operating device while touching the screen (that is, continuously changes the designated position). In such a case, the designated position indicated by the first coordinate information and the second coordinate information will be different. That is, the processing unit 140 can also accept a drag operation, a flick operation, or other designations of positions that change continuously over time made by the user of the first terminal 10.

[0044] The generation means 144 includes a first generation means 1441 and a second generation means 1442, and generates information necessary for remote operation to be supplied to the second terminal 20 according to the operation made by the first terminal 10.

[0045] Specifically, when the pointer is at the initial position (in other words, when it is determined that no designation is made at the position in a series of operations such as a drag operation or a flick operation, and a touch operation is newly or independently performed), the first generation means 1441 generates first operation information for moving the pointer on the second terminal 20 to the position on the partner screen corresponding to the designated position, based on the first coordinate information and the reference position information indicating the reference position coordinates. Specifically, the first coordinate information is information representing the relative coordinates of the position indicated by the first coordinate information with respect to the reference position indicated by the reference position information. As a specific example, it is expressed as an amount obtained by converting the vector amount in the coordinate system of the screen of the first terminal 10, with the position indicated by the reference position information as the starting point and the position indicated by the first coordinate information as the end point, into the vector amount in the coordinate system of the screen of the second terminal 20 using the display area mapping information MAP. That is, the first coordinate information corresponds to relative position information viewed from the initial position regarding the touch position on the partner screen. For example, when the configurations and usage modes of the first terminal 10 and the second terminal 20 are as shown in FIG. 3A, as shown in FIG. 4, first operation information indicating F(X1 - Xo, Y1 - Yo), which is the amount of movement from the reference position, is generated based on the acquired first coordinate information (X1, Y1).

[0046] On the other hand, if the pointer is not in its initial position (in other words, if the position is specified in the middle of a series of operations such as a drag operation or a flick operation), the first coordinate information indicates the difference (amount of position change) between the coordinates of the position previously specified on the screen of the first terminal 10 and the coordinates of the currently specified position, converted into a difference on the screen of the second terminal 20 using the display area mapping information MAP. For example, as shown in Figure 5, if P2 is specified following the specification of position P1 from the state shown in Figure 4, first operation information is generated based on the acquired first coordinate information (X2, Y2) that indicates the amount of movement F (ΔX, ΔY) on the screen of the second terminal 20 that corresponds to the amount of movement (amount of position change) on the screen of the first terminal 10. Here, ΔX = X2 - X1 and ΔY = Y2 - Y1.

[0047] The second generation means 1442 generates second operation information to move the pointer to the initial position. As an example, the second operation information indicates the maximum negative value of the movement amount that the second terminal 20 can receive. Generally, if the value (movement amount) input from the mouse is an amount that moves the pointer to a position outside the screen, control is performed to display the pointer at a predetermined position such as the corner of the screen. If the second terminal 20 employs such control, supplying the movement amount (-x_max, -y_max) to the second terminal 20 will move the pointer to the left corner of the screen. Here, -x_max and -y_max indicate the maximum values ​​of the movement amount in the x and y directions, respectively. For example, as shown in Figure 6, if the specification is released at position P3, second operation information indicating the movement amount (-x_max, -y_max) is generated based on the second coordinate information (X3, Y3).

[0048] As another example, the second operation information is information representing relative coordinates with respect to a reference position indicated by reference position information with respect to the position indicated by the second coordinate information. As a specific example, it may be an amount obtained by converting a vector amount in the coordinate system of the screen of the first terminal 10, which consists of two coordinate components with the position indicated by the second coordinate information as the starting point and the position indicated by the reference position information as the ending point, into the coordinate system of the screen of the second terminal 20. In this case, similar to the first operation information, the second operation information becomes relative position information. Alternatively, the amount indicated by the second operation information may not be the maximum value of minus or the amount that exactly returns to the initial position as described above, as long as it is an amount for moving at least to a position corresponding to outside the screen from the position where the current pointer is displayed.

[0049] The transmitting means 145 transmits the first operation information generated by the first generating means 1441 or the second operation information generated by the second generating means 1442 to the second terminal 20 using the communication unit 110 according to the operation content. Specifically, when a position is specified (such as when the finger of the user of the first terminal 10 touches the screen), the first operation information is transmitted, while when the position specification is canceled (such as when the finger of the user of the first terminal 10 leaves the screen), the second operation information is transmitted.

[0050] When the second terminal 20 receives the first operation information, it moves the pointer MP on the display screen according to the received first operation information.

[0051] For example, in FIG. 4, assume that the position indicated by P1 on the left side of FIG. 4 is specified as the designated position. In this case, the first operation information representing a vector with the position indicated by the reference position information as the starting point and the position of the asterisk as the ending point is transmitted from the first terminal 10 to the second terminal 20. When the second terminal 20 receives the first operation information, it moves the pointer MP on the display screen to p1, which is the position corresponding to P1, as shown on the right side of FIG. 4, according to the received first operation information. Since the first operation information corresponds to relative position information seen from the initial position regarding the touch position on the other party's screen, even for the second terminal 20 where the information that can be received as information representing the user's operation content is limited to relative position information, the pointer can be moved without problems.

[0052] An example of the operation of the first terminal 10 will be explained using Figure 7. The processing unit 140, which is operating according to program PR1, will perform the processing illustrated in Figure 7 until it is instructed to terminate the execution of program PR1. When the processing unit 140 detects that a new position has been specified, such as when a finger touches the screen (S202; YES), it generates first operation information and generates and transmits first operation information to move the pointer to the position on the other side's screen relative to the specified position on the screen of the first terminal 10 (S204). If the specification is released at that position (S206; YES), in other words, if a tap operation (click operation) is performed at that position (S206; YES), second operation information is generated and transmitted, and the pointer is returned to its initial position (S210). As a result, the screen reflecting the tap operation performed by the user is displayed on the display unit of the second terminal 20 (and the first terminal 10 which is mirroring it). On the other hand, if the touch position is updated (S206: NO and S211: YES), the first operation information is generated and transmitted (S212). As a result, the screen reflecting the drag operation performed by the user is displayed on the display unit of the second terminal 20 (and the first terminal 10 which is mirroring it). Subsequently, if a touch release is detected (S206: YES), since the touch position and the release position are different (S208: NO), the second operation information is generated and transmitted, and the pointer is returned to its initial position (S210). As a result, the series of drag operations are reflected on the display unit of the second terminal 20 (and the first terminal 10) (S214).

[0053] The following describes the movement of the pointer in the second terminal 20 that receives the first and second operation information. Although the following description refers to the screen displayed on the second terminal 20, due to mirroring, the same screen content is reproduced on the display unit of the first terminal 10 in virtually real time.

[0054] Figure 8 shows the movement of the pointer when a tap operation is performed. As shown in Figure 8(a), when the second terminal 20 receives the first operation information generated in S204 of Figure 7, it performs a drawing process to move the pointer MP from its initial position (in this case, set to the left corner of the screen) to the specified position. Subsequently, as shown in Figure 8(b), when the second operation information generated in S208 of Figure 7 is received at this position, the second terminal 20 performs a drawing process to change the appearance of the point, such as making the pointer MP light up, in order to inform the user of the second terminal 20 that a tap (click operation) has been performed. The drawing process when the second operation information is received may be omitted. When the second operation information generated in S208 of Figure 7 is received, a drawing process is performed to move the pointer back to its initial position.

[0055] The pointer drawing process on the second terminal 20 when a drag operation is performed will be explained using Figures 9 and 10. Figure 9 shows the processes on the first terminal 10 and the second terminal 20 in chronological order. Figure 10 shows the screen of the second terminal 20 in each process. First, at time t1, when the user touches a certain position on the screen, first operation information corresponding to that position is transmitted (t2). Upon receiving this first information, the second terminal 20 starts drawing processing to move the pointer (t3). At this time, the pointer is displayed as shown in Figures 10(a) and (b). In this state, as the user moves their finger while touching the screen, new first operation information is generated at predetermined timings (t4), and the second terminal 20 updates the display position of the pointer according to each piece of first operation information (t5 and Figure 10(c)). When the user of the first terminal 10 lifts their finger from the screen at t5 (t6), second operation information is generated (t7), and drawing processing based on the second operation information is performed (t8). Specifically, as shown in Figure 9(d), the pointer is displayed at that position, and then moved back to its initial position. It is preferable that, unlike tapping, no special drawing process is performed when the finger is lifted. This clearly distinguishes it from tapping.

[0056] As described above, according to this embodiment, the screen of the second terminal 20 is mirrored, and operation information reflecting the specified position on the screen of the first terminal is transmitted to the second terminal 20. As a result, even if the second terminal 20 does not accept absolute coordinates, the first terminal 10 that remotely controls the second terminal 20 can reflect touch operations that specify absolute positions in the movement of the pointer on the second terminal 20. In other words, according to this embodiment, it becomes possible to remotely control a target terminal that only accepts relative position information using an external input device that outputs absolute positions.

[0057] <Other Embodiments> Although one embodiment of the present invention has been described above, this embodiment can be modified as follows.

[0058] In the above embodiment, there was only one second terminal 20 connected to the first terminal 10, but multiple other terminals, including the second terminal 20, may be connected to the first terminal 10 simultaneously. When multiple other terminals, including the second terminal 20, are connected to the first terminal 10 simultaneously, the processing unit 140 sets up multiple partitioned screen display areas in the display area of ​​the display device included in the input / output unit 120 that correspond one-to-one with the display screen areas of each of the multiple other terminals connected to its own terminal. The processing unit 140 then determines a reference position for each of the multiple partitioned screen areas that corresponds to the initial position on the corresponding other terminal.

[0059] In addition to the first terminal 10, a second partner terminal is connected to the first terminal 10, which displays an operation object on the display screen at a position based on the absolute position of the input operator. The first terminal 10 may display image data received from the second partner terminal on its own display screen. For example, as shown in Figure 11, the first terminal 10 is connected to the second terminal 20 as well as a device 90 that accepts absolute position input. The first terminal 10 calculates and transmits to the device 90 the position on the screen of the device 90 corresponding to the position specified on the screen of the first terminal 10, according to the display area mapping information MAP, as the first operation information and the second operation information described above. In other words, the process of converting absolute position to relative position (movement amount) is not performed. For example, when connecting to the second terminal 20 or device 90, the identification information of the second terminal 20 or 90 is received from the partner terminal, and it is determined whether the first operation information and the second operation information are defined by absolute position or relative position.

[0060] In this case, the processing unit 140 may display a predetermined object at the position on the display screen of its own terminal indicated by the first coordinate information. If the position on the display screen of its own terminal indicated by the first coordinate information is within the display area corresponding to the other terminal, the object is displayed in the first mode. If the position on the display screen of its own terminal indicated by the first coordinate information is within the display area corresponding to the second other terminal, the object is displayed in the second mode, which is different from the first mode. A specific example of the second mode is a mode in which a circle of a predetermined size lights up for a predetermined time at the touch position (hereinafter referred to as the normal display mode). A specific example of the first mode is a mode in which a circle of a predetermined size lights up for a longer time than the predetermined time at the touch position. According to this mode, it is possible to determine whether the operating terminal is a terminal that only accepts relative positions (mouse input) or a terminal that can accept absolute positions through the display mode of the object.

[0061] In particular, during the process in which the pointer returns to its initial position after the completion of the above-mentioned operation, there is a possibility that the users of the first terminal 10 and the second terminal 20 may be given a feeling of afterimage. Especially when the second terminal 20 and the third terminal 90 are connected simultaneously, the feeling of feedback of the operations performed by the user to the drawing content differs depending on whether the second terminal 20 is remotely controlled or the device 90 is remotely controlled. Therefore, the user of the first terminal 10 may become confused as to whether they actually manipulated the pointer to return to its initial position or simply performed a tap operation. From this perspective, it is expected that the user's discomfort can be reduced by changing the appearance and presentation (for example, shape, size, color, movement, or changes in these over time) of the pointer displayed on the screen of the first terminal 10, depending on whether the pointer is displayed on the screen corresponding to the display screen of the second terminal 20 or on the screen corresponding to the display screen of the device 90. In the example shown in Figure 11, the shape of the pointer C1 displayed in area R5 is round, while the shape of the pointer C2 displayed in area R6 is star-shaped.

[0062] Thus, the pointer drawn independently by the first terminal 10 may be superimposed on the screen of the terminal connected to the first terminal 10. Alternatively, the first terminal 10 may not display any pointers based on user operations, but instead display the screen including the pointer image that is displayed on the terminal connected to the first terminal 10. In this case, the pointer drawn independently by the first terminal 10 and the pointer image included in the image data received from the other terminal will not be displayed twice.

[0063] In short, the present invention relates to a terminal that connects to a remote terminal that displays an operation object at a position on its display screen based on the amount of position change of an input operator, and displays image data received from the remote terminal on the display screen of the local terminal, and the terminal is configured to perform the following steps: a first acquisition step of acquiring first coordinate information indicating a position on the local terminal's display screen, which has been specified by the user of the local terminal; a first generation step of generating first operation information for moving the operation object from an initial position on the remote terminal's display screen to a position on the remote terminal's screen corresponding to the specified position; a first transmission step of transmitting the first operation information to the remote terminal; a second acquisition step of acquiring second coordinate information indicating a position on the local terminal's display screen, which has been released from the user's specification; a second generation step of generating second operation information for moving the operation object to the initial position; and a second transmission step of transmitting the second operation information to the remote terminal.

[0064] 1...Computer system, 10...First terminal, 20...Second terminal, 110...Communication unit, 120...Input / output unit, 140...Processing unit, 141...Display control means, 142...Location information acquisition means, 143...Decision means, 144...Generation means, 1441...First generation means, 1442...Second generation means, 145...Operation information transmission means, 160...Attitude detection unit, 130...Storage unit, MAP...Display area mapping information

Claims

1. A program that causes the computer of a terminal that connects to a remote terminal that displays an operation object at a position on its display screen based on the amount of position change of an input operator, and displays image data received from the remote terminal on the remote terminal's display screen, to execute: a first acquisition step of acquiring first coordinate information indicating a position on the remote terminal's display screen as specified by the user of the remote terminal; a first generation step of generating first operation information for moving the operation object from an initial position on the remote terminal's display screen to a position on the remote terminal's screen corresponding to the specified position, based on the first coordinate information acquired in the first acquisition step; a first transmission step of transmitting the first operation information to the remote terminal; a second acquisition step of acquiring second coordinate information indicating a position on the remote terminal's display screen as specified by the user, after the user's specification has been released; a second generation step of generating second operation information for moving the operation object from a position displayed on the remote terminal's screen to the initial position based on the first operation information, in response to acquiring the second coordinate information; and a second transmission step of transmitting the second operation information to the remote terminal.

2. The program according to claim 1, characterized in that the computer further performs a determination step to determine a reference position coordinate on the display screen of its terminal that corresponds to the initial position, and in the first generation step, generates the first operation information based on the relative coordinates of the first coordinate information with respect to the reference position coordinate.

3. The program according to claim 1 or 2, which, when the local terminal connects to a computer simultaneously with a plurality of other terminals including the other terminal, includes the steps of setting a partitioned screen area on the local terminal's display screen that corresponds to the display screen area of ​​the other terminal, and determining a reference position within the partitioned screen area that corresponds to the initial position in the first generation step.

4. The terminal is connected to a second partner terminal that displays an operation object at a position on a display screen based on the absolute position of an input operator, and is capable of displaying image data received from the second partner terminal on the display screen of the terminal, the program further causes the computer to display a predetermined object at the position on the display screen of the terminal indicated by the first coordinate information, the predetermined object is displayed in a first manner if the position on the display screen of the terminal indicated by the first coordinate information is within a display area corresponding to the partner terminal, and is displayed in a second manner different from the first manner if the position on the display screen of the terminal indicated by the first coordinate information is within a display area corresponding to the second partner terminal, the program according to claim 1.

5. A terminal comprising: means for connecting to a partner terminal that displays an operation object at a position on its display screen based on the amount of position change of an input operator; means for displaying image data received from the partner terminal on the display screen of the own terminal; first acquisition means for acquiring first coordinate information indicating a position on the display screen of the own terminal as specified by the user; first generation means for generating first operation information for moving the operation object from an initial position on the display screen of the partner terminal to a position on the screen of the partner terminal corresponding to the specified position, based on the first coordinate information acquired by the first acquisition means; first transmission means for transmitting the first operation information to the partner terminal; second acquisition means for acquiring second coordinate information indicating a position on the display screen of the own terminal as specified by the user; second generation means for generating second operation information for moving the operation object from a position displayed on the screen of the partner terminal to the initial position based on the first operation information in response to acquiring the second coordinate information; and second transmission means for transmitting the second operation information to the partner terminal.

6. A computer system comprising a terminal and a partner terminal that displays an operation object at a position on its display screen based on the amount of position change of an input operator, wherein the terminal includes means for displaying image data received from the partner terminal on its own display screen, first acquisition means for acquiring first coordinate information indicating a position on the display screen of the terminal specified by a user, first generation means for generating first operation information for moving the operation object from an initial position on the display screen of the partner terminal to a position on the screen of the partner terminal corresponding to the specified position, based on the first coordinate information acquired by the first acquisition means, first transmission means for transmitting the first operation information to the partner terminal, second acquisition means for acquiring second coordinate information indicating a position on the display screen of the terminal after the user's specification has been released, second generation means for generating second operation information for moving the operation object from a position displayed on the screen of the partner terminal to the initial position based on the first operation information in response to acquiring the second coordinate information, and second transmission means for transmitting the second operation information to the partner terminal.

Images