The technology of this application relates to a
software engine that enables a user to interact with a display (e.g., television) by using an
image capture device (e.g., camera). In one non-limiting example, a user can operate a device (such as a
mobile phone) to run an application that interfaces with the
software engine, so that the device's camera can operate in
unison with a separate display to enable unique types of user interactions. For example, the user may use such an application to tap a
user interface button or draw a shape with their hand, directly on the surface of a larger display (e.g., television), in view of the
mobile phone's camera; the application's visual state would update in response to the user's captured gestures and to the position and orientation of the
mobile phone. In another non-limiting example, a user, while playing a game utilizing the
software engine, swings a device (such as a mobile phone) in
free space, to control the position and orientation of a
paddle within the three dimensional scene of the game, presented on the larger display.