Systems and methods for tracking a search target within a larger context in a camera live view

Abstract

An example method includes receiving a user query to search for a target in a live camera preview of a scene. The method includes detecting whether the target is present in a first frame of a plurality of frames of the live preview. The detecting comprises applying a detection algorithm on at least a larger context of the scene in the first frame. The method includes, responsive to detecting that the target is present in the first frame: pausing the detection algorithm on frames successive to the first frame, predicting a relative location of the detected target in the larger context based on a previously determined location of the detected target relative to the larger context, and initiating a tracking of the detected target in the plurality of frames based on the predicted relative location of the detected target. The method includes providing, in the live preview, the tracked target.

Description

SYSTEMS AND METHODS FOR TRACKING A SEARCH TARGET WITHIN A LARGER CONTEXT IN A CAMERA LIVE VIEWCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No.63 / 761,381, filed February 21, 2025, which is incorporated herein by reference in its entirety.BACKGROUND

[0002] Object detection and tracking algorithms can be used to track objects in a camera live view. These algorithms can be applied to various types of objects, including text and other visual elements. The efficiency of tracking can vary depending on the size and complexity of the object being tracked. A magnifier application may be used to magnify a tracked object.SUMMARY

[0003] This application generally relates to a live search in a camera live view, and is designed to operate as a tool that allows users to view finer details in a live preview of a scene, as well as precisely locate relevant information in the scene. Matched text may be highlighted in the camera live stream view in real time. This can be challenging since by the time an image is processed via optical character recognition (OCR), the live stream may have changed. The present application uses a live tracker to infer a matched text location of a current view from OCR information of a past view. The live tracker may be based on a homography matrix that tracks a portion of the text by understanding its transformation from an initial frame to a current frame. These algorithms can be suitably adapted to be applied to various types of targets, including text and other visual elements. In some scenarios, it may be more efficient to track a larger contextual object to infer the location of a smaller object within that context.

[0004] Performing a detection algorithm, such as an object detection algorithm, or an OCR operation, on every frame can consume considerable computational resources, and may also result in higher latency. As the camera live view may be changing, the higher latency may also cause errors in detection and / or tracking of the target. However, tracking a larger context in the scene (e.g., a large paragraph, a large drawing, a street scene, etc.) may be performed in a computationally efficient manner. In particular, when a smaller target has a relatively and substantially fixed position relative to the larger contextual object, simpler mapping algorithms may be used to predict the location of the smaller target within thetracked larger contextual object. Accordingly, it is more advantageous to perform the detection algorithm in a first frame to detect the target, pause the detection algorithm in successive frames, and track the predicted location of the smaller target within the tracked larger contextual object. Such an approach can result in a higher stability of the tracking, a higher accuracy of the search, reduce latency, and also consume fewer computational resources.

[0005] At times, users may need assistance in sorting through large amounts of text and visual information to find details, a task that demands a high cognitive load, especially for low vision users. A magnifier application (app) can enable users to view details they may otherwise not see by enlarging and enhancing the world around them via their camera. The app’s tools may assist users in finding what they are looking for in their surroundings, and personalize the live or still image to bring such details into focus. For some users, the magnifier app may be a situational tool (e.g., when they may have forgotten their reading or distance glasses, or when they are doing detailed work in jewelry, soldering, or needlework). For other users with more moderate to severe low vision, the magnifier app may be a more permanent tool that replaces a physical magnifying glass or spyglass.

[0006] In one aspect, a computer-implemented method is provided. The method includes receiving a user query to search for a portion of text in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames. The method also includes detecting, by the image capturing device, whether the portion of text in the received user query is present in a first frame of the plurality of frames, wherein the detecting comprises performing an optical character recognition (OCR) operation on at least a larger portion of text in the first frame. The method further includes, responsive to detecting that the portion of text is present in the first frame: pausing the OCR operation on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected portion of text in the larger portion of text, wherein the predicting is based on a previously determined location of the detected portion of text in the larger portion of text in the first frame, and initiating a tracking of the detected portion of text in the frames successive to the first frame based on the predicted relative location of the detected portion of text, wherein the tracking of the detected portion of text is based on tracking the larger portion of text and the previously determined location of the detected portion of text within the larger portion of text. The method additionally includes providing, in the live preview comprising the frames successive to the first frame, the tracked portion of text.

[0007] In another aspect, a system is provided. The system may include one or more processors. The system may also include data storage, where the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the system to carry out operations. The operations may include receiving a user query to search for a portion of text in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames. The operations may also include detecting, by the image capturing device, whether the portion of text in the received user query is present in a first frame of the plurality of frames, wherein the detecting comprises performing an optical character recognition (OCR) operation on at least a larger portion of text in the first frame. The operations may additionally include, responsive to detecting that the portion of text is present in the first frame: pausing the OCR operation on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected portion of text in the larger portion of text, wherein the predicting is based on a previously determined location of the detected portion of text in the larger portion of text in the first frame, and initiating a tracking of the detected portion of text in the frames successive to the first frame based on the predicted relative location of the detected portion of text, wherein the tracking of the detected portion of text is based on tracking the larger portion of text and the previously determined location of the detected portion of text within the larger portion of text. The operations may also include providing, in the live preview comprising the frames successive to the first frame, the tracked portion of text.

[0008] In another aspect, a computing device is provided. The device includes a primary camera and a secondary camera that share a common field of view. The device also includes one or more processors and data storage that has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out operations. The operations may include receiving a user query to search for a portion of text in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames. The operations may also include detecting, by the image capturing device, whether the portion of text in the received user query is present in a first frame of the plurality of frames, wherein the detecting comprises performing an optical character recognition (OCR) operation on at least a larger portion of text in the first frame. The operations may additionally include, responsive to detecting that the portion of text is present in the first frame: pausing the OCR operation on frames successive to the first frame, predicting, in frames successive to the first frame, a relativelocation of the detected portion of text in the larger portion of text, wherein the predicting is based on a previously determined location of the detected portion of text in the larger portion of text in the first frame, and initiating a tracking of the detected portion of text in the frames successive to the first frame based on the predicted relative location of the detected portion of text, wherein the tracking of the detected portion of text is based on tracking the larger portion of text and the previously determined location of the detected portion of text within the larger portion of text. The operations may also include providing, in the live preview comprising the frames successive to the first frame, the tracked portion of text.

[0009] In another aspect, an article of manufacture is provided. The article of manufacture may include a non-transitory computer-readable medium having stored thereon program instructions that, upon execution by one or more processors of a computing device, cause the computing device to carry out operations. The operations may include receiving a user query to search for a portion of text in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames. The operations may also include detecting, by the image capturing device, whether the portion of text in the received user query is present in a first frame of the plurality of frames, wherein the detecting comprises performing an optical character recognition (OCR) operation on at least a larger portion of text in the first frame. The operations may additionally include, responsive to detecting that the portion of text is present in the first frame: pausing the OCR operation on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected portion of text in the larger portion of text, wherein the predicting is based on a previously determined location of the detected portion of text in the larger portion of text in the first frame, and initiating a tracking of the detected portion of text in the frames successive to the first frame based on the predicted relative location of the detected portion of text, wherein the tracking of the detected portion of text is based on tracking the larger portion of text and the previously determined location of the detected portion of text within the larger portion of text. The operations may also include providing, in the live preview comprising the frames successive to the first frame, the tracked portion of text.

[0010] In another aspect, a program is provided. The program upon execution by one or more processors of a computing device, causes the computing device to carry out operations. The operations may include receiving a user query to search for a portion of text in a live preview of a scene being captured by an image capturing device, wherein thelive preview comprises a plurality of frames. The operations may also include detecting, by the image capturing device, whether the portion of text in the received user query is present in a first frame of the plurality of frames, wherein the detecting comprises performing an optical character recognition (OCR) operation on at least a larger portion of text in the first frame. The operations may additionally include, responsive to detecting that the portion of text is present in the first frame: pausing the OCR operation on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected portion of text in the larger portion of text, wherein the predicting is based on a previously determined location of the detected portion of text in the larger portion of text in the first frame, and initiating a tracking of the detected portion of text in the frames successive to the first frame based on the predicted relative location of the detected portion of text, wherein the tracking of the detected portion of text is based on tracking the larger portion of text and the previously determined location of the detected portion of text within the larger portion of text. The operations may also include providing, in the live preview comprising the frames successive to the first frame, the tracked portion of text.

[0011] In another aspect, a computer-implemented method is provided. The method includes receiving a user query to search for a target in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames. The method also includes detecting, by the image capturing device, whether the target is present in a first frame of the plurality of frames, wherein the detecting comprises applying a detection algorithm on at least a larger context of the scene in the first frame. The method further includes, responsive to detecting that the target is present in the first frame: pausing the detection algorithm on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected target in the larger context of the scene, wherein the predicting is based on a previously determined location of the detected target relative to the larger context, and initiating a tracking of the detected target in the frames successive to the first frame based on the predicted relative location of the detected target, wherein the tracking of the detected target is based on tracking the larger context and the previously determined location of the detected target relative to the larger context. The method additionally includes providing, in the live preview comprising the frames successive to the first frame, the tracked target.

[0012] In another aspect, a system is provided. The system may include one or more processors. The system may also include data storage, where the data storage has storedthereon computer-executable instructions that, when executed by the one or more processors, cause the system to carry out operations. The operations may include receiving a user query to search for a target in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames. The operations may also include detecting, by the image capturing device, whether the target is present in a first frame of the plurality of frames, wherein the detecting comprises applying a detection algorithm on at least a larger context of the scene in the first frame. The operations may additionally include, responsive to detecting that the target is present in the first frame: pausing the detection algorithm on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected target in the larger context of the scene, wherein the predicting is based on a previously determined location of the detected target relative to the larger context, and initiating a tracking of the detected target in the frames successive to the first frame based on the predicted relative location of the detected target, wherein the tracking of the detected target is based on tracking the larger context and the previously determined location of the detected target relative to the larger context. The operations may also include providing, in the live preview comprising the frames successive to the first frame, the tracked target.

[0013] In another aspect, a computing device is provided. The device includes a primary camera and a secondary camera that share a common field of view. The device also includes one or more processors and data storage that has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out operations. The operations may include receiving a user query to search for a target in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames. The operations may also include detecting, by the image capturing device, whether the target is present in a first frame of the plurality of frames, wherein the detecting comprises applying a detection algorithm on at least a larger context of the scene in the first frame. The operations may additionally include, responsive to detecting that the target is present in the first frame: pausing the detection algorithm on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected target in the larger context of the scene, wherein the predicting is based on a previously determined location of the detected target relative to the larger context, and initiating a tracking of the detected target in the frames successive to the first frame based on the predicted relative location of the detected target, wherein the tracking of the detected target is based on tracking the larger context and thepreviously determined location of the detected target relative to the larger context. The operations may also include providing, in the live preview comprising the frames successive to the first frame, the tracked target.

[0014] In another aspect, an article of manufacture is provided. The article of manufacture may include a non-transitory computer-readable medium having stored thereon program instructions that, upon execution by one or more processors of a computing device, cause the computing device to carry out operations. The operations may include receiving a user query to search for a target in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames. The operations may also include detecting, by the image capturing device, whether the target is present in a first frame of the plurality of frames, wherein the detecting comprises applying a detection algorithm on at least a larger context of the scene in the first frame. The operations may additionally include, responsive to detecting that the target is present in the first frame: pausing the detection algorithm on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected target in the larger context of the scene, wherein the predicting is based on a previously determined location of the detected target relative to the larger context, and initiating a tracking of the detected target in the frames successive to the first frame based on the predicted relative location of the detected target, wherein the tracking of the detected target is based on tracking the larger context and the previously determined location of the detected target relative to the larger context. The operations may also include providing, in the live preview comprising the frames successive to the first frame, the tracked target.

[0015] In another aspect, a program is provided. The program upon execution by one or more processors of a computing device, causes the computing device to carry out operations. The operations may include receiving a user query to search for a target in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames. The operations may also include detecting, by the image capturing device, whether the target is present in a first frame of the plurality of frames, wherein the detecting comprises applying a detection algorithm on at least a larger context of the scene in the first frame. The operations may additionally include, responsive to detecting that the target is present in the first frame: pausing the detection algorithm on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected target in the larger context of the scene, wherein the predicting is based on a previously determined location of the detected target relative to thelarger context, and initiating a tracking of the detected target in the frames successive to the first frame based on the predicted relative location of the detected target, wherein the tracking of the detected target is based on tracking the larger context and the previously determined location of the detected target relative to the larger context. The operations may also include providing, in the live preview comprising the frames successive to the first frame, the tracked target.

[0016] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the figures and the following detailed description and the accompanying drawings.BRIEF DESCRIPTION OF THE FIGURES

[0017] Figure 1 is an illustration of front, right-side, and rear views of a digital camera device 100, in accordance with example embodiments.

[0018] Figure 2 illustrates an example search in live camera view, in accordance with example embodiments.

[0019] Figure 3 illustrates another example textual search in live camera view, in accordance with example embodiments.

[0020] Figure 4 illustrates an example homography, in accordance with example embodiments.

[0021] Figure 5 illustrates an image processing pipeline, in accordance with example embodiments.

[0022] Figure 6 is an example table illustrating an OCR pipeline for image frame processing, in accordance with example embodiments.

[0023] Figure 7 illustrates an example magnifier application, in accordance with example embodiments.

[0024] Figure 8 illustrates examples of live tracking, in accordance with example embodiments.

[0025] Figure 9 is a block diagram of an example computing device, in accordance with example embodiments.

[0026] Figure 10 is a flowchart of a method, in accordance with example embodiments.

[0027] Figure 11 is a flowchart of a method, in accordance with example embodiments.DETAILED DESCRIPTION

[0028] Example methods, devices, and systems are described herein. It should be understood that the words “example” and “exemplary” are used herein to mean “serving as an example, instance, or illustration.” Any embodiment or feature described herein as being an “example” or “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or features. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein.

[0029] Thus, the example embodiments described herein are not meant to be limiting. Aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

[0030] Further, unless context suggests otherwise, the features illustrated in each of the figures may be used in combination with one another. Thus, the figures should be generally viewed as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are necessary for each embodiment.Overview

[0031] In many instances, a user may want to locate an object and / or text in a live preview. For example, a user may want to locate a specific product in a busy store (e.g. a pharmacy or a grocery aisle), search for a specific point of interest (e.g., a street sign, a specific airline on an airline arrival or departure information screen, a restroom), find a certain item on a page of a multi-page document (e.g., a restaurant menu, a service manual, a book), seek instructions on three-dimensional (3D) packaging, including curved packaging, locate a specific title at a library or bookstore, locate labels and / or specific patterns in a complex illustration (e.g., a drawing, an architectural drawing, a product design), and so forth. Other use cases may include assisting visually challenged users. Generally speaking, a user may enter a query, receive a visual and / or haptic confirmation when live search is performed, when results are received, and / or when no results are found. For example, a user may be notified as long as a matching result is present in the preview frame.

[0032] Existing approaches to live search in a camera live view apply the OCR operation to every image frame. However, the OCR operation takes time. By the time OCR is completed and a location of a matched text in a camera frame is located from a fraction of a second ago, the actual text location in the current camera live view may have already moved, leading to errors. Furthermore, a continuous running of the OCR operation cancause the power consumption to be extremely high (~8W). Another challenge is that existing techniques use a tracker to directly track the matched text. However, when the matched text comprises a small number of characters in a large amount of dense text, the tracking can become unstable, resulting in errors and / or a diminished user experience.

[0033] Accordingly, there is a need for techniques that can improve the tracking of a matched text while also reducing the power consumption due to the OCR. The techniques described herein achieve that by performing an OCR on a first frame, determining a location of the matched text relative to a larger amount of text, tracking the larger amount of text from frame to frame while predicting a location of the matched text based on the location of the matched text relative to the larger amount of text. For example, under the assumption that the location of characters in paragraphs are relatively stable, this solution obtains the homography matrix by tracking a large area of the paragraph (tracking targets). As another example, the locations of labels on a drawing are relatively stable, and this solution obtains the homography matrix by tracking the drawing (tracking targets). The homography matrix represents the target's transformation from the initial frame to the current frame. It can be applied to a matched text’ s location in the paragraph (output targets) to predict the updated location in the current frame. Also, it can be applied to a location of a label in the drawing (output targets) to predict the updated location of the label in the current frame. While homography is mentioned as an example of a tracking method, other tracking methods, including those that may utilize different sensor technologies or algorithms for tracking larger contextual objects versus smaller target objects, are also contemplated. For instance, different tracking technologies could be used for tracking a street sign versus tracking text on the street sign.

[0034] These as well as other aspects, advantages, and alternatives will become apparent to those reading the following description, with reference where appropriate to the accompanying drawings. Further, it should be understood that the discussion in this overview and elsewhere in this document is provided by way of example only and that numerous variations are possible.Example Camera Systems

[0035] As image capture devices, such as cameras, become more popular, they may be employed as standalone hardware devices or integrated into various other types of devices. For instance, still and video cameras are now regularly included in wireless computing devices (e.g., mobile devices, such as mobile phones), tablet computers, laptop computers,video game interfaces, home automation devices, and even automobiles and other types of vehicles.

[0036] The physical components of a camera may include one or more apertures through which light enters, one or more recording surfaces for capturing the images represented by the light, and lenses positioned in front of each aperture to focus at least part of the image on the recording surface(s). The apertures may be of a fixed size or may be adjustable. In an analog camera, the recording surface may be a photographic film. In a digital camera, the recording surface may include an electronic image sensor (e.g., a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) sensor) to transfer and / or store captured images in a data storage unit (e.g., memory).

[0037] One or more shutters may be coupled to, or positioned near, the lenses or the recording surfaces. Each shutter may either be in a closed position, in which it blocks light from reaching the recording surface, or an open position, in which light is allowed to reach the recording surface. The position of each shutter may be controlled by a shutter button. For instance, a shutter may be in the closed position by default. When the shutter button is triggered (e.g., pressed), the shutter may change from the closed position to the open position for a period of time, known as the shutter cycle. During the shutter cycle, an image may be captured on the recording surface. At the end of the shutter cycle, the shutter may change back to the closed position.

[0038] Alternatively, the shuttering process may be electronic. For example, before an electronic shutter of a CCD image sensor is “opened,” the sensor may be reset to remove any residual signal in its photodiodes. While the electronic shutter remains open, the photodiodes may accumulate charge. When or after the shutter closes, these charges may be transferred to longer-term data storage. Combinations of mechanical and electronic shuttering may also be possible.

[0039] Regardless of type, a shutter may be activated and / or controlled by input or other trigger that is not a shutter button. For instance, the shutter may be activated by a softkey, a gesture, audio (such as a voice command), a timer, detected motion, or some other trigger. Herein, the term “capture” may refer to any mechanical and / or electronic shuttering process that results in one or more images being recorded, regardless of how the shuttering process is triggered or controlled.

[0040] The exposure of a captured image may be determined by a combination of the size of the aperture, the brightness of the light entering the aperture, and the length of the shutter cycle (also referred to as the shutter length, the exposure length, or the exposure time).Additionally, a digital and / or analog gain (e.g., based on an ISO setting) may be applied to the image, thereby influencing the exposure. In some embodiments, the term “exposure length,” “exposure time,” or “exposure time interval” may refer to the shutter length multiplied by the gain for a particular aperture size. Thus, these terms may be used somewhat interchangeably, and should be interpreted as possibly being a shutter length, an exposure time, and / or any other metric that controls the amount of signal response that results from light reaching the recording surface.

[0041] In some implementations or modes of operation, a camera may capture one or more still images each time image capture is triggered. In other implementations or modes of operation, a camera may capture a video image by continuously capturing images at a particular rate (e.g., 24 frames per second) as long as image capture remains triggered (e.g., while the shutter button is held down). Some cameras, when operating in a mode to capture a still image, may open the shutter when the camera device or application is activated, and the shutter may remain in this position until the camera device or application is deactivated. While the shutter is open, the camera device or application may capture and display a representation of a scene on a viewfinder (sometimes referred to as displaying a “preview frame”). When image capture is triggered, one or more distinct payload images of the current scene may be captured.

[0042] Cameras, including digital and analog cameras, may include software to control one or more camera functions and / or settings, such as aperture size, exposure time, gain, and so on. Additionally, some cameras may include software that digitally processes images during or after image capture. While the description above refers to cameras in general, it may be particularly relevant to digital cameras. Digital cameras may be standalone devices (e.g., a DSLR camera) or may be integrated with other devices.

[0043] Either or both of a front-facing camera and a rear-facing camera may include or be associated with an ambient light sensor (ALS) that may continuously or from time to time determine the ambient brightness of a scene that the camera can capture. In some devices, the ALS can be used to adjust the display brightness of a screen associated with the camera (e.g., a viewfinder). When the determined ambient brightness is high, the brightness level of the screen may be increased to make the screen easier to view. When the determined ambient brightness is low, the brightness level of the screen may be decreased, also to make the screen easier to view as well as to potentially save power. Additionally, the ambient light sensor’s input may be used to determine an exposure time of an associated camera, or to help in this determination.

[0044] Figure 1 is an illustration of front, right-side, and rear views of a digital camera device 100, in accordance with example embodiments. Digital camera device 100 may be, for example, a mobile device (e.g., a mobile phone), a tablet computer, or a wearable computing device. However, other embodiments are possible. Digital camera device 100 may include various elements, such as a body 102, a front-facing camera 104, a multielement display 106, a shutter button 108, and other buttons 110. Digital camera device 100 could further include one or more rear-facing cameras 112, 114. Front-facing camera 104 may be positioned on a side of body 102 typically facing a user while in operation, or on the same side as multi-element display 106. Rear-facing cameras 112, 114 may be positioned on a side of body 102 opposite front-facing camera 104. Referring to the cameras as front-facing and rear-facing is arbitrary, and digital camera device 100 may include multiple cameras positioned on various sides of body 102.

[0045] Multi-element display 106 could represent a cathode ray tube (CRT) display, a light-emitting diode (LED) display, a liquid crystal display (LCD), a plasma display, or any other type of display known in the art. In some embodiments, multi-element display 106 may display a digital representation of the current image being captured by front-facing camera 104 and / or rear-facing cameras 112, 114, or an image that could be captured or was recently captured by either or both of these cameras. Thus, multi-element display 106 may serve as a viewfinder for either camera. Multi -element display 106 may also support touchscreen and / or presence-sensitive functions that may be able to adjust the settings and / or configuration of any aspect of digital camera device 100.

[0046] Multi-element display 106 may include additional features related to a camera application. For example, multiple modes may be available for a user, including, a motion mode, portrait mode, video mode, video bokeh mode, and so forth. The camera application may be in camera mode and provide additional features, such as a reverse icon to activate reverse camera view, a trigger button to capture a previewed image, and a photo stream icon to access a database of captured images. Also for example, a magnification ratio slider may be displayed and a user can move a virtual object along the magnification ratio slider to select a magnification ratio. In some embodiments, a user may use the multi-element display 106, also referred to herein as the display screen, to adjust the magnification ratio (e.g., by moving two fingers on display screen in an outward motion away from each other), and magnification ratio slider may automatically display the magnification ratio.

[0047] Front-facing camera 104 may include an image sensor and associated optical elements such as lenses. Front-facing camera 104 may offer zoom capabilities or couldhave a fixed focal length. In other embodiments, interchangeable lenses could be used with front-facing camera 104. Front-facing camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. Front-facing camera 104 also could be configured to capture still images, video images, or both. Further, front-facing camera 104 could represent a monoscopic, stereoscopic, or multiscopic camera. Rear-facing cameras 112, 114 may be similarly or differently arranged. Additionally, front-facing camera 104, rear-facing cameras 112, 114, or both, may be an array of one or more cameras.

[0048] Either or both of front-facing camera 104 and rear-facing cameras 112, 114 may include or be associated with an illumination component that provides a light field to illuminate a target object. For instance, an illumination component could provide flash or constant illumination of the target object (e.g., using one or more LEDs). An illumination component could also be configured to provide a light field that includes one or more of structured light, polarized light, and light with specific spectral content. Other types of light fields known and used to recover three-dimensional (3D) models from an object are possible within the context of the embodiments herein.

[0049] In some digital camera devices 100, either or both of front-facing camera 104 and rear-facing cameras 112, 114 may include or be associated with an ambient light sensor that may continuously or from time to time determine the ambient brightness of a scene that the camera can capture. In some devices, the ambient light sensor can be used to adjust the display brightness of a screen associated with the camera (e.g., a viewfinder). When the determined ambient brightness is high, the brightness level of the screen may be increased to make the screen easier to view. When the determined ambient brightness is low, the brightness level of the screen may be decreased, also to make the screen easier to view as well as to potentially save power. Additionally, the ambient light sensor’s input may be used to determine an exposure time of an associated camera, or to help in this determination.

[0050] Digital camera device 100 could be configured to use multi-element display 106 and either front-facing camera 104 or rear-facing cameras 112, 114 to capture images of a target object (e.g., a subject within a scene). The captured images could be a plurality of still images or a video image (e.g., a series of still images captured in rapid succession with or without accompanying audio captured by a microphone). The image capture could be triggered by activating shutter button 108, pressing a softkey on multi-element display 106, or by some other mechanism. Depending upon the implementation, the images could be captured automatically at a specific time interval, for example, upon pressing shutter button108, upon appropriate lighting conditions of the target object, upon moving digital camera device 100 a predetermined distance, or according to a predetermined capture schedule.

[0051] As noted above, the functions of digital camera device 100 (or another type of digital camera) may be integrated into a computing device, such as a wireless computing device, cell phone, tablet computer, laptop computer, and so on. For example, a camera controller may be integrated with the digital camera device 100 to control one or more functions of the digital camera device 100.Example Methods for Live Tracking

[0052] Figure 2 illustrates an example search 200 in live camera view, in accordance with example embodiments. Device 205 may be configured to display a live camera view 210 of a scene. For example, a user may be scanning the aisle of a pharmacy for a particular product. The user may enter a query 215 to indicate the product to be searched for. Upon performing a successful search, the display may highlight the product (e.g., by placing a bounding box around the product). In some embodiments, the product may be displayed using a highlight feature to distinguish it from other objects in the live camera view 210 of the scene. For example, a border of the bounding box may have additional color mixing with the background color to provide a high contrast (e.g., even on different colored backgrounds). In some embodiments, the color for a search results area may be increased in contrast to make an identified text more prominent.

[0053] In some embodiments, as the user walks with the camera, the live camera view changes and the highlighted product may be tracked live. In some embodiments, after the target (e.g., product) is found, the user interface (UI) design may be configured to reduce the background brightness to highlight the target. Also, for example, an input functionality may be provided for switching on / off background effect to viewing of surrounding information. As indicated, the target may be marked with a bounding box 220 (e.g., a square frame with rounded comers). In the event multiple targets are identified, key areas of live camera view 210 may be consolidated (e.g., to avoid overly cluttered highlighted regions).

[0054] In some embodiments, the receiving of the user query comprises receiving a textual input. For example, query 215 may be a textual input entered by the user. In some embodiments, the receiving of the user query comprises receiving a voice input. For example, the user may interact with device 205 using voice commands. In some embodiments, the voice command may be transcribed to a textual input. In someembodiments, the receiving of the user query comprises receiving an image input. For example, the user may input an image of the product to be searched for.

[0055] In some embodiments, the receiving of the user query comprises receiving the user query in a language other than English. For example, query 215 may be in English, or in one or more additional languages recognized by device 205. In some embodiments, the voice commands may be in English, or another language other than English.

[0056] Figure 3 illustrates an example textual search 300 in live camera view, in accordance with example embodiments. Device 305 may be configured to display a live camera view 310. A user may be viewing a larger portion of text 315 as displayed in the live camera view 310. In some embodiments, the larger portion of text 315 may be in English, and / or in a language other than English.

[0057] In some embodiments, the larger portion of text comprises a number of words that exceeds a threshold number. For example, a larger portion of text 315 may be a large, dense paragraph including many words. In some aspects, the font size may be small. The user may enter “uf ’ as a search query 320. Generally, when the matched text (e.g., “uf ’) includes a smaller number of characters in a larger amount of dense text (e.g., larger portion of text 315), existing tracking approaches can become unstable, resulting in errors and / or a diminished user experience. As illustrated, one or more appearances of the matched text “ut” 325 may be found in the larger portion of text 315. As previously described, the matched text “uf ’ 325 may be tracked in the live camera view 310.

[0058] Figure 4 illustrates an example homography, in accordance with example embodiments. For example, when the user enters a search query “ut,” the device may detect whether the word “uf ’ in the received user query is present in a first frame of a plurality of frames. The detecting comprises performing an optical character recognition (OCR) operation on at least a larger portion of text in the first frame. Also, for example, responsive to detecting that the word “ut” is present in the first frame, the device may be configured to pause the OCR operation on frames successive to the first frame. The device may be configured to predict, in frames successive to the first frame, a relative location of the detected portion of text in the larger portion of text, wherein the predicting is based on a previously determined location of the detected portion of text in the larger portion of text in the first frame. For example, the larger portion of text 405 is shown. A mapping may be established that maps the larger portion of text 405 in the first frame to the larger portion of text 410 in the frame successive to the first frame.

[0059] Some embodiments involve determining an inter-frame image transformation of the larger portion of text between the first frame and a second frame successive to the first frame, and wherein the predicting may be based on the inter-frame image transformation. For example, the mapping of the larger portion of text 405 in the first frame to the larger portion of text 410 in the frame successive to the first frame may be based on an inter-frame image transformation. In some embodiments, the inter-frame image transformation may be a homography matrix. For example, a homography matrix may be determined for a large paragraph in transitioning from the first frame to the second frame, and the homography matrix may then be applied to successive texts in successive frames to predict a location of the matched text in the successive frames. This makes the tracking more stable.

[0060] Subsequently, a relative location of the detected word “uf ’ in the larger portion of text 410 may be determined, and used to predict the relative location of the detected word “ut” in the portions of textual content 420 in frames successive to the first frame. For example, the inter-frame image transformation may be applied to the detected word “uf ’ in frames successive to the first frame. Accordingly, the device may be configured to initiate a tracking of the detected portion of text in the frames successive to the first frame based on the predicted relative location of the detected portion of text, wherein the tracking of the detected portion of text is based on tracking the larger portion of text and the previously determined location of the detected portion of text within the larger portion of text.

[0061] Figure 5 illustrates an image processing pipeline 500, in accordance with example embodiments. Some embodiments involve receiving a user query to search for a portion of text in a live preview of a scene being captured by an image capturing device. The live preview may include a plurality of frames. Each frame of the plurality of frames may be associated with image metadata. For example, image frame 505 of the plurality of frames may be associated with image metadata 510. Some embodiments involve, responsive to the receiving of the user query, decreasing a frame rate for the image capturing device. For example, when the user query is received and a search is initiated, the camera’s frame rate may be decreased (e.g., to 15 frames per second) so that fewer image frames are generated.

[0062] In some embodiments, image processing pipeline 500 may include one or more pipelines. For example, an OCR pipeline may perform OCR related operations, and a tracker pipeline may use the information from the OCR pipeline to perform image processing and object tracking related operations.

[0063] In some embodiments, a camera application programming interface (API) may receive image frame 505 and image metadata 510 that may include data related to imageframe 505, such as, for example, image size, resolution, pixel-wise color values, timestamp, location data, and so forth. Classifier 515 may estimate whether the image frame includes text. In the event the image frame does not include text, the OCR operation may not be performed. Some embodiments involve, responsive to the receiving of the user query, converting the plurality of frames to grayscale images. For example, image converter 520 may convert the image frame 505 to a grayscale image. Some embodiments involve, responsive to the receiving of the user query, lowering an image resolution for the plurality of frames. For example, image converter 520 may convert the image frame 505 to a lower resolution image. In some embodiments, a new stream configuration (e.g., YUV 420 888 + 1280 * 720) may be added to the camera pipeline in addition to an existing configuration (e.g., YUV_420_888 + 4032 * 2268) for better streaming resolution. In some embodiments, the new stream configuration may be attached when live OCR is active, and may be detached when live OCR is no longer active. For purposes of the image processing pipeline 500, image frame 505 may be processed as a lower resolution, and / or grayscale image (e.g., to conserve compute resources).

[0064] Some embodiments involve detecting, by the image capturing device, whether the portion of text in the received user query is present in a first frame of the plurality of frames, wherein the detecting comprises performing an optical character recognition (OCR) operation on at least a larger portion of text in the first frame. For purposes of this illustration, image frame 505 may be a first frame to be processed by the OCR gate 525. For example, OCR gate 525 may receive data from the classifier 515 and the image converter 520. OCR gate 525 may determine which image frames of the plurality of frames are to be provided to the OCR pipeline. For example, when the first frame is processed by the OCR pipeline, successive frames may not be entered into the OCR pipeline. Accordingly, OCR may not be applied on every image frame, resulting in considerable savings for compute resources.

[0065] Some embodiments involve detecting, by the image capturing device, whether the portion of text in the received user query is present in a first frame of the plurality of frames. The detecting may include performing an OCR operation on at least a larger portion of text in the first frame. For example, OCR applier 530 may apply OCR to the first frame and provide the output to the OCR detection list generator 535. Generally speaking, OCR detection list generator 535 generates a list of all textual content in a past frame (e.g., the first frame) that has been identified for possible tracking along with the respectivelocations. In some aspects, the OCR operation may not be performed when no text is detected in an image frame.

[0066] Some embodiments involve, responsive to detecting that the portion of text is present in the first frame, pausing the OCR operation on frames successive to the first frame. For example, upon providing the output of the OCR detection list generator 535 to the magnifier tracker 540, the OCR operation may be paused and the OCR pipeline may remain idle until OCR gate 525 allows a next image frame to enter the OCR pipeline.

[0067] In some embodiments, the pausing of the OCR operation comprises pausing the OCR operation until a next application of the OCR operation at a next frame in the plurality of frames. Such embodiments involve applying the OCR operation at the next frame. For example, OCR gate 525 may identify a next frame on which to apply the OCR operation, and provide a downscaled version of the next frame to OCR applier 530.

[0068] In some embodiments, the next application of the OCR operation may be based on a rate of application of the OCR operation. For example, the rate of application of the OCR operation may be a fixed, periodic time interval (e.g., every 1 or 1.5 seconds). In some embodiments, the rate of application of the OCR operation may be based on an analysis of the scene. For example, a scene with more than a threshold amount of text, or a scene that changes quickly, the rate of application of the OCR operation may be increased. Similarly, where less optical character recognition is needed (e.g., the camera pans slowly, the textual content in the scene is low and / or remains substantially the same), the rate of application of the OCR operation may be decreased. Generally, determining the rate of application of the OCR operation involves a trade-off between power savings from pausing the OCR operation, and latency -related artifacts and / or errors (e.g., missing new information entering a live camera view by not performing the OCR operation) from not performing the OCR operation often enough. In some embodiments, the OCR gate 525 may be configured to apply the rate of application of the OCR operation. As described, this rate may be a predetermined rate, or may be dynamically determined based on scene analysis. In some situations, the rate of application of the OCR operation limits the OCR model in a most effective way as the general power consumption may drop from approximately 8W to 3.6W, a 55% reduction. Also, for example, the power consumption of the live view OCR pipeline may drop from approximately 5W to 0.7W, a reduction of about 86%.

[0069] The tracker pipeline receives the output of the OCR pipeline. For example, the output of the OCR detection list generator 535 may be provided to the magnifier tracker 540, which also receives the output from the image converter 520 (e.g., a lower resolution,grayscale image of a current frame). The magnifier tracker 540 stores the information received from the OCR detection list generator 535 about a past image frame (e.g., the first image frame) and uses this information to predict the location of the matched text in successive frames. Detection list matcher 545 matches the user query with results of the OCR operation as provided in the list generated by OCR detection list generator 535. Results accumulator 550 determines the text and location to be tracked, and output results 555 may involve the highlighted text in the live camera view in response to the user query.

[0070] In the absence of the magnifier tracker 540, and by applying the OCR operation on every frame, the text highlights may be slow to flow in live view. This may be caused by a longer time taken by OCR processing (e.g., 0.5 to 1 second), so that by the time one image frame is processed by the OCR operation, the camera live view may have changed. Also, a continuous running of the OCR operation may cause the power consumption to be significantly high.

[0071] Figure 6 is an example table 600 illustrating an OCR pipeline for image frame processing, in accordance with example embodiments. Table 600 includes columns labeled 6C1 to 6C6 and rows labeled 6R1 to 6R8.

[0072] Row 6R1 and column 6C1 indicate that a first frame, referred to as Frame 1, is processed by the image processing pipeline (e.g., image processing pipeline 500 of Figure 5). In column 6C2 and row 6R1, the input to the OCR pipeline is Frame 1. Also, as indicated in column 6C2 and rows 6R2, 6R3, and 6R4, the OCR pipeline is rate limited. This means that the OCR pipeline is running at a limited rate. The same process may be repeated in each cycle. For example a second processing cycle is illustrated in column 6C2 and rows 6R5 to 6R8.

[0073] Column 6C3 illustrates the status of the OCR pipeline. For example, when Frame 1 enters the OCR pipeline, it may be processed. During the processing of Frame 1, successive frames numbered 2 to A are not processed. TAs indicated in row 6R3, the processing of Frame 1 may be completed between the arrival of frames numbered A and A+l. Accordingly, as indicated in row 6R4 and column 6C3, the OCR pipeline remains idle, and power savings may be realized. The same process may be repeated in each cycle. For example a second processing cycle is illustrated in column 6C3 and rows 6R5 to 6R8.

[0074] Column 6C4 illustrates an input for the OCR tracker. As indicated in rows 6R1 and 6R2 (resp. 6R5 and 6R6), there is no output as Frame 1 (resp. Frame B+l) is being processed by the OCR pipeline. Between frames A and A+l, the processing of Frame 1 is completed, and column 6C4 indicates that the OCR results from Frame 1 are provided, andthe tracker stores this information for tracking purposes. Subsequently, for frames A+l to B, the OCR pipeline remains idle, and there is no corresponding OCR result, as indicated in column 6C4 and row 6R4.

[0075] The same process may be repeated in each cycle. For example a second processing cycle is illustrated in column 6C4 and rows 6R5 to 6R8. For example, as row 6R7 and column 6C4 indicates, the OCR results from frame B+l may be received, the tracker may store this information, and may discard the stored information from a previous cycle (e.g., stored information related to Frame 1).

[0076] Columns 6C5 and 6C6 illustrate the tracker image input and output. For example, with reference to rows 6R1 and 6R2, there is no OCR result output, and accordingly, there is no OCR information. The tracker image input is frames 1 to A, but the tracker image output is empty.

[0077] With reference to row 6R3, between frames A to A+l, the tracker image input is empty, and accordingly there is no tracker output.

[0078] Generally speaking, there is a distinction between the tracker output being “empty” and the tracker output being “no output.” For example, for frames 1 to A, there is no OCR information. Accordingly, the tracker does not have information as to the location of the matched text. So the tracker output is empty. In this case, the frames 1 to A are displayed without bounding boxes, highlights, background dimming, etc. However, when there is no image input, as between frames A and A+l, then there is no tracker output. In this case, there is no image to display. Frames A+l to C, the tracker output is based on the OCR information from Frame 1, and a predicted location of the matched text in frames 2 to C. Accordingly, the matched text may be displayed with a bounding bow, the bounding box may be highlighted, and / or the image background may be dimmed to make the matched text appear more prominent.

[0079] With reference to rows 6R4, 6R5, and 6R6, the stored information related to Frame 1 is available to the tracker. In this case, the tracker image input includes frames A+l to C, and the tracker output includes the predicted matched text locations in frames A+l to C, where the matched text locations are based on the OCR results of Frame 1. As previously described, the matched text locations may be inferred based on an inter-frame image transformation (e.g., a homography matrix) of the larger portion of text between the first frame such as Frame 1 and a second frame successive to the first frame, such as Frame 2.

[0080] Generally speaking, the homography matrix may be a 3 X 3 matrix that maps pixel coordinates on one frame with the pixel coordinates of another frame. For purposes of thisapplication, it may be challenging to track matched text with very few characters, especially if the matched text appears in a much larger body of textual content. For example, a general tracker typically tracks a target in one view, and then uses a tracking algorithm such as Lucas-Kanade method to estimate the location of the target in a new view. Such approaches focus on the tracking targets and their estimated location. However, it can be challenging to track a few characters among dense texts if we treat characters as tracking targets directly. Applying a tracking algorithm may result in unstable and / or erroneous tracking. However, it may be less challenging to track the much larger body of textual content. Under the assumption that the relative position of the matched text within the much larger body of textual content remains unchanged, the homography matrix can be applied to predict the relative location of the matched text from frame to frame. The homography matrix represents the target's transformation from the initial frame to the current frame. It can be applied to a character’s location in the paragraph (output targets) to obtain the updated location in the current frame.

[0081] In some embodiments, the values for A, B, C, and D may be chosen as A = 5, B = 10, C = 15, andD = 20.

[0082] Although the technology is described with reference to an OCR operation performed on a large body of text, similar techniques may be applied in a more general setting, such as to locate a target in a larger context (e.g., to predict a location of other objects). For example, a large and / or complicated drawing (e.g., an architectural drawing, electrical circuit drawing) may involve several components and / or labels. The techniques described herein may be used to locate and track a specific label and / or a smaller component of the drawing. Also, for example, in manufacturing (e.g., airline, ships, spacecraft, automobiles, etc.), labels and / or smaller components may be located and tracked within very large portions of the component being manufactured.

[0083] As described previously, similar techniques may be applied to spot a specific product in a busy store (e.g. a pharmacy or a grocery aisle), look for a specific point of interest (e.g., a street sign, a specific airline on an airline arrival or departure information screen, a restroom sign), locate a certain item on a page of a multi-page document (e.g., a restaurant menu, a service manual, a book), seek instructions on 3D (including curved) packaging, locate a specific title at a library or bookstore, locate labels on a complex illustration (a drawing, an architectural drawing, a product design), and so forth. In general, the techniques can be applied to track any smaller target within a larger, more easily trackable context.

[0084] In some embodiments, the display may be configured to provide directions to a physical location of the matched text. For example, the user may be directed toward a book in the library by providing instructions such as “continue walking down this aisle” and as they get closer to the book, the user may be provided with instructions such as “look to your right, the book is placed on the top shelf, and is the third book from the left, next to the book with the red cover titled “X”.”

[0085] Additional examples applications may include finding a medication (e.g., using a name of the medication, an image related to a container for the medication, etc.) on a busy pharmacy shelf, locating a specific book title at a library or bookstore, sorting through mail to find items addressed to a particular person or address, searching for a street number after exiting a taxi, flipping through a multi-page menu to find a specific section like “desserts.”

[0086] Referring again to Figure 6, the “OCR pipeline” of columns 6C2 and 6C3 may be replaced with a “detection pipeline,” and the “tracker OCR result input” of column 6C4 can be replaced with a “tracker result input.” The OCR related operations may be replaced with detection related operations, and the “matched text” may be replaced with a “matched target.” Accordingly, frame 1 may be processed by a detection algorithm, frames 2 to A may be received while frame 1 is being processed. The tracker output for frames 1 to A may be empty as there is no result available for the detection algorithm. Also, between frames 2 and A+l, the frame rate may be limited. The processing of frame 1 for the target may be completed between frames A and A+l, and the tracker may store this result. As there is no tracker image output between frames A and A+l, there is no tracker output.

[0087] The detection pipeline may be paused between frames A+l to B, resulting in power savings. However, for frames A+l to C, a location of the matched target may be inferred by predicting a relative location of the matched target within a larger context, where the larger context may be tracked in a stable and efficient manner.

[0088] As column 6C2 indicates, the processing of frame 1 for detection (e.g., OCR or object detection) may continue while frames 2 to A are received. Accordingly, in the event every frame were to be processed directly for detection, as existing approaches do, then the processing of frame 2 will likely begin after frame A is received. This can result in higher use of computational resources, higher latency, and a reduced accuracy of tracking.

[0089] Upon detecting a matched text, the user interface (UI) design may be configured to reduce the background brightness to highlight the matched text. In some embodiments, the UI may display a virtual button to switch the background effect on or off in order to viewportions of the image near the matched text. Also, for example, the matched text may be marked with a square frame (e.g., with rounded corners).

[0090] Figure 7 illustrates an example magnifier application, in accordance with example embodiments. Device 705 may include a live camera view 710. The user query 715 may be to search for “micro.” An enlarged view 720 of the live camera view 710 displays a bounding box 725 that highlights a region of interest. In some embodiments, a magnified view 730 of the image within bounding box 725 may be displayed by device 705, where the matched text 735 (e.g., “micro”) is displayed with a bounding box.

[0091] In some embodiments, the UI may be configured to dynamically adjust a thickness of the bounding box 725. For example, when the matched text is small, the UI may dynamically thicken the border to ensure that users can see the position clearly. As the text enlarges, the UI may be configured to adjust the border to an appropriate proportion for users to conveniently view the content.

[0092] In some embodiments, the UI may enable animation (e.g., at the edges of magnified view 730) to indicate to the user when the search is initiated, ongoing, and / or completed. For example, a red colored edge may be used to indicate no match (e.g., no match for the user query 715 is found), and a green colored edge may be used to indicate a match (e.g., matched text “micro” has been located). In some embodiments, the UI may be configured so that the edges have a flickering effect to draw the user’s attention. Also, for example, the text border may be configured to include additional color mixing with the background color (e.g., to ensure a high contrast even on different colored backgrounds). For example, the color for the search results area may be increased in contrast to make the text more prominent.

[0093] Figure 8 illustrates examples of live tracking, in accordance with example embodiments. For example, image 805 illustrates an example where two matched texts 810 are detected and tracked corresponding to a user query for “tran.” Image 815 illustrates an example where multiple search results 820 are located and tracked corresponding to a user query for “col.” As illustrated, the key areas may be consolidated to avoid overly cluttered highlighted regions.

[0094] Other use cases may include helping visually challenged users. For example, users may enter a query (e.g., provide a voice input), receive a visual and / or haptic confirmation when live search is performed, when results are received, and / or when no results are found. For example, a user may be notified as long as a matching result is present in the preview frame. The notifications may be visual, haptic, audible, spoken, and so forth. The devicemay be configured to enable the user to give voice input to start a search. Also, for example, the device may be configured to vibrate when the search is initiated, and / or provide near continuous vibration when a positive match is detected.

[0095] Live search, as described herein, may be initiated by a voice search. In some embodiments, a virtual button may be displayed in the viewfinder that can initiate the search. For example, tapping the virtual button may open a keyboard to facilitate entry of a user query. Also, for example, holding the button down may enable voice entry without providing the keyboard. When voice entry is active, the user may be provided an indication that voice entry is active and what the user says appears in the search box as text. Also, for example, releasing the virtual button can initiate the search. The live search may be terminated by the user by selecting an exit button on the UI. In some embodiments, the live search may be timed out after a certain time to protect system health. The UI may provide the user an ability to search for an exact phrase (e.g., by adding quotes or « » on either side of the phrase).Example Computing Device Architectures

[0096] Figure 9 is a block diagram of an example computing device 900, in accordance with example embodiments. In particular, computing device 900 shown in Figure 9 can be configured to perform at least one function described herein, including method 1000.

[0097] Computing device 900 may include a user interface module 901, a network communications module 902, one or more processors 903, data storage 904, one or more cameras 918, one or more sensors 920, and power system 922, all of which may be linked together via a system bus, network, or other connection mechanism 905.

[0098] User interface module 901 can be operable to send data to and / or receive data from external user input / output devices. For example, user interface module 901 can be configured to send and / or receive data to and / or from user input devices such as a touch screen, a computer mouse, a keyboard, a keypad, a touch pad, a trackball, a joystick, a voice recognition module, and / or other similar devices. User interface module 901 can also be configured to provide output to user display devices, such as one or more cathode ray tubes (CRT), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, either now known or later developed. User interface module 901 can also be configured to generate audible outputs, with devices such as a speaker, speaker jack, audio output port, audio output device, earphones, and / or other similar devices. User interface module 901 canfurther be configured with one or more haptic devices that can generate haptic outputs, such as vibrations and / or other outputs detectable by touch and / or physical contact with computing device 900. In some examples, user interface module 901 can be used to provide a graphical user interface (GUI) for utilizing computing device 900.

[0099] Network communications module 902 can include one or more devices that provide one or more wireless interfaces 907 and / or one or more wireline interfaces 908 that are configurable to communicate via a network. Wireless interface(s) 907 can include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth™ transceiver, a Zigbee® transceiver, a Wi-Fi™ transceiver, a WiMAX™ transceiver, an LTE™ transceiver, and / or other type of wireless transceiver configurable to communicate via a wireless network. Wireline interface(s) 908 can include one or more wireline transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or similar transceiver configurable to communicate via a twisted pair wire, a coaxial cable, a fiber-optic link, or a similar physical connection to a wireline network.

[0100] In some examples, network communications module 902 can be configured to provide reliable, secured, and / or authenticated communications. For each communication described herein, information for facilitating reliable communications (e.g., guaranteed message delivery) can be provided, perhaps as part of a message header and / or footer (e.g, packet / message sequencing information, encapsulation headers and / or footers, size / time information, and transmission verification information such as cyclic redundancy check (CRC) and / or parity check values). Communications can be made secure (e.g, be encoded or encrypted) and / or decrypted / decoded using one or more cryptographic protocols and / or algorithms, such as, but not limited to, Data Encryption Standard (DES), Advanced Encryption Standard (AES), a Rivest- Shamir- Adelman (RSA) algorithm, a Diffie-Hellman algorithm, a secure sockets protocol such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or Digital Signature Algorithm (DSA). Other cryptographic protocols and / or algorithms can be used as well or in addition to those listed herein to secure (and then decry pt / decode) communications.

[0101] One or more processors 903 can include one or more general purpose processors (e.g., central processing unit (CPU), etc.), and / or one or more special purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). One or moreprocessors 903 can be configured to execute computer-readable instructions 906 that are contained in data storage 904 and / or other instructions as described herein.

[0102] Data storage 904 can include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of one or more processors 903. The one or more computer-readable storage media can include volatile and / or nonvolatile storage components, such as optical, magnetic, organic or other memory or disc storage, which can be integrated in whole or in part with at least one of one or more processors 903. In some examples, data storage 904 can be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disc storage unit), while in other examples, data storage 904 can be implemented using two or more physical devices.

[0103] Data storage 904 can include computer-readable instructions 906 and perhaps additional data. In some examples, data storage 904 can include storage required to perform at least part of the herein-described methods, scenarios, and techniques and / or at least part of the functionality of the herein-described devices and networks. In particular, computer-readable instructions 906 can include instructions that, when executed by processor(s) 903, enable computing device 900 to provide for some or all of the functionality described herein.

[0104] In some embodiments, computer-readable instructions 906 can include instructions that, when executed by processor(s) 903, enable computing device 900 to carry out operations. In some embodiments, the operations may include receiving a user query to search for a portion of text in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames. The operations may also include detecting, by the image capturing device, whether the portion of text in the received user query is present in a first frame of the plurality of frames, wherein the detecting comprises performing an optical character recognition (OCR) operation on at least a larger portion of text in the first frame. The operations may additionally include, responsive to detecting that the portion of text is present in the first frame: pausing the OCR operation on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected portion of text in the larger portion of text, wherein the predicting is based on a previously determined location of the detected portion of text in the larger portion of text in the first frame, and initiating a tracking of the detected portion of text in the frames successive to the first frame based on the predicted relative location of the detected portion of text, wherein the tracking of the detected portion of text is basedon tracking the larger portion of text and the previously determined location of the detected portion of text within the larger portion of text. The operations may also include providing, in the live preview comprising the frames successive to the first frame, the tracked portion of text.

[0105] In some embodiments, the operations may include receiving a user query to search for a target in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames. The operations may also include detecting, by the image capturing device, whether the target is present in a first frame of the plurality of frames, wherein the detecting comprises applying a detection algorithm on at least a larger context of the scene in the first frame. The operations may additionally include, responsive to detecting that the target is present in the first frame: pausing the detection algorithm on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected target in the larger context of the scene, wherein the predicting is based on a previously determined location of the detected target relative to the larger context, and initiating a tracking of the detected target in the frames successive to the first frame based on the predicted relative location of the detected target, wherein the tracking of the detected target is based on tracking the larger context and the previously determined location of the detected target relative to the larger context. The operations may also include providing, in the live preview comprising the frames successive to the first frame, the tracked target

[0106] The computing device 900 may include a live tracking module 912 operable to detect and track a target object in a live camera view. Computing device 900 may provide the tracking in the live view. Live tracking module 912 may be a software application or subsystem within computing device 900 that is operable to receive one or more image frames, perform an OCR operation as described herein, detect a matched text corresponding to a user query, track the matched text in successive frames based on a previous location in a previous frame, and provide the tracked matched text with appropriate UI features. In some implementations, live tracking module 912 may receive the one or more image frames and associated metadata from camera(s) 918.

[0107] In some examples, computing device 900 can include one or more cameras 918. Camera(s) 918 can include one or more image capture devices, such as still and / or video cameras, equipped to capture light and record the captured light in one or more images; that is, camera(s) 918 can generate image(s) of captured light. The one or more images can be one or more still images and / or one or more images utilized in video imagery.Camera(s) 918 can capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other frequencies of light. Camera(s) 918 can include a wide camera, a tele camera, an ultrawide camera, and so forth. Also, for example, camera(s) 918 can be front-facing or rear-facing cameras with reference to computing device 900. Camera(s) 918 can include camera components such as, but are not limited to, an aperture, shutter, recording surface (e.g., photographic film and / or an image sensor), lens, and / or shutter button. The camera components may be controlled at least in part by software executed by one or more processors 903.

[0108] In some examples, camera(s) 918 could be oriented at a specific rotation angle and may capture image frames at that rotation angle (also referred to herein as a lend position). In some implementations, the rotation angle is a horizontal angle. That is, the rotation angle may be the horizontal rotation of camera(s) 918 from an initial pointing direction. In other implementations, the rotation angle is a vertical angle. That is, the rotation angle may be the vertical rotation of camera(s) 918 from an initial pointing direction. In example embodiments, the initial pointing direction may correspond to the pointing direction of camera(s) 918 as it captures a first image frame in a stream of image frames.

[0109] In some examples, computing device 900 can include one or more sensors 920. Sensors 920 can be configured to measure conditions within computing device 900 and / or conditions in an environment of computing device 900 and provide data about these conditions. For example, sensors 920 can include one or more of: (i) sensors for obtaining data about computing device 900, such as, but not limited to, a thermometer for measuring a temperature of computing device 900, a battery sensor for measuring power of one or more batteries of power system 922, and / or other sensors measuring conditions of computing device 900; (ii) an identification sensor to identify other objects and / or devices, such as, but not limited to, a Radio Frequency Identification (RFID) reader, proximity sensor, one-dimensional barcode reader, two-dimensional barcode (e.g., Quick Response (QR) code) reader, and a laser tracker, where the identification sensors can be configured to read identifiers, such as RFID tags, barcodes, QR codes, and / or other devices and / or object configured to be read and provide at least identifying information; (iii) sensors to measure locations and / or movements of computing device 900, such as, but not limited to, a tilt sensor, a gyroscope, an accelerometer, a Doppler sensor, a GPS device, a sonar sensor, a radar device, a laser-displacement sensor, and a compass; (iv) an environmental sensor to obtain data indicative of an environment of computing device 900, such as, but not limitedto, an infrared sensor, an optical sensor, a light sensor (e.g., an ambient light sensor), a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a movement sensor, a microphone, a sound sensor, an ultrasound sensor and / or a smoke sensor; and / or (v) a force sensor to measure one or more forces (e.g., inertial forces and / or G-forces) acting about computing device 900, such as, but not limited to one or more sensors that measure: forces in one or more dimensions, torque, ground force, friction, and / or a zero moment point (ZMP) sensor that identifies ZMPs and / or locations of the ZMPs. Many other examples of sensors 920 are possible as well.

[0110] Power system 922 can include one or more batteries 924 and / or one or more external power interfaces 926 for providing electrical power to computing device 900. Each battery of the one or more batteries 924 can, when electrically coupled to the computing device 900, act as a source of stored electrical power for computing device 900. One or more batteries 924 of power system 922 can be configured to be portable. Some or all of one or more batteries 924 can be readily removable from computing device 900. In other examples, some or all of one or more batteries 924 can be internal to computing device 900, and so may not be readily removable from computing device 900. Some or all of one or more batteries 924 can be rechargeable. For example, a rechargeable battery can be recharged via a wired connection between the battery and another power supply, such as by one or more power supplies that are external to computing device 900 and connected to computing device 900 via the one or more external power interfaces. In other examples, some or all of one or more batteries 924 can be non-rechargeable batteries.

[0111] One or more external power interfaces 926 of power system 922 can include one or more wired-power interfaces, such as a USB cable and / or a power cord, that enable wired electrical power connections to one or more power supplies that are external to computing device 900. One or more external power interfaces 926 can include one or more wireless power interfaces, such as a Qi wireless charger, that enable wireless electrical power connections, such as via a Qi wireless charger, to one or more external power supplies. Once an electrical power connection is established to an external power source using one or more external power interfaces 926, computing device 900 can draw electrical power from the external power source the established electrical power connection. In some examples, power system 922 can include related sensors, such as battery sensors associated with the one or more batteries or other types of electrical power sensors.

[0112] One or more external power interfaces 926 of power system 922 can include one or more wired-power interfaces, such as a USB cable and / or a power cord, that enablewired electrical power connections to one or more power supplies that are external to computing device 900. One or more external power interfaces 926 can include one or more wireless power interfaces, such as a Qi wireless charger, that enable wireless electrical power connections, such as via a Qi wireless charger, to one or more external power supplies. Once an electrical power connection is established to an external power source using one or more external power interfaces 926, computing device 900 can draw electrical power from the external power source the established electrical power connection. In some examples, power system 922 can include related sensors, such as battery sensors associated with the one or more batteries or other types of electrical power sensors.Example Methods of Operation

[0113] Figure 10 is a flowchart of a method, in accordance with example embodiments. Method 1000 may include various blocks or steps. The blocks or steps may be carried out individually or in combination. The blocks or steps may be carried out in any order and / or in series or in parallel. Further, blocks or steps may be omitted or added to method 1000.

[0114] The blocks of method 1000 may be carried out by various elements of computing device 900 as illustrated and described in reference to Figure 9.

[0115] Block 1010 involves receiving a user query to search for a portion of text in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames.

[0116] Block 1020 involves detecting, by the image capturing device, whether the portion of text in the received user query is present in a first frame of the plurality of frames, wherein the detecting comprises performing an optical character recognition (OCR) operation on at least a larger portion of text in the first frame.

[0117] Block 1030 involves responsive to detecting that the portion of text is present in the first frame: pausing the OCR operation on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected portion of text in the larger portion of text, wherein the predicting is based on a previously determined location of the detected portion of text in the larger portion of text in the first frame, and initiating a tracking of the detected portion of text in the frames successive to the first frame based on the predicted relative location of the detected portion of text, wherein the tracking of the detected portion of text is based on tracking the larger portionof text and the previously determined location of the detected portion of text within the larger portion of text.

[0118] Block 1040 involves providing, in the live preview comprising the frames successive to the first frame, the tracked portion of text.

[0119] Some embodiments involve, responsive to the receiving of the user query, decreasing a frame rate for the image capturing device.

[0120] Some embodiments involve, responsive to the receiving of the user query, lowering an image resolution for the plurality of frames.

[0121] Some embodiments involve, responsive to the receiving of the user query, converting the plurality of frames to grayscale images.

[0122] In some embodiments, the larger portion of text comprises a number of portion of texts that exceeds a threshold number.

[0123] In some embodiments, the pausing of the OCR operation comprises pausing the OCR operation until a next application of the OCR operation at a next frame in the plurality of frames. Such embodiments involve applying the OCR operation at the next frame. In some embodiments, the next application of the OCR operation may be based on a rate of application of the OCR operation. In some embodiments, the rate of application of the OCR operation may be based on an analysis of the scene.

[0124] In some embodiments, the receiving of the user query comprises receiving a textual input.

[0125] In some embodiments, the receiving of the user query comprises receiving a voice input.

[0126] In some embodiments, the receiving of the user query comprises receiving the user query in a language other than English.

[0127] In some embodiments, the larger portion of text is in a language other than English.

[0128] Some embodiments involve detecting that the portion of text is not present in the first frame. Such embodiments involve not performing the OCR operation on the first frame.

[0129] Some embodiments involve performing the OCR operation on another larger portion of text in another frame of the plurality of frames.

[0130] In some embodiments, the providing of the tracked portion of text comprises displaying an enlarged view of a portion of the live preview that comprises the tracked portion of text.

[0131] In some embodiments, the providing of the tracked portion of text comprises highlighting the tracked portion of text.

[0132] In some embodiments, the providing of the tracked portion of text comprises reducing a screen brightness level for a portion of the live preview outside the tracked portion of text.

[0133] Some embodiments involve determining an inter-frame image transformation of the larger portion of text between the first frame and a second frame successive to the first frame, and wherein the predicting may be based on the inter-frame image transformation.

[0134] In some embodiments, the inter-frame image transformation may be a homography matrix.

[0135] In some embodiments, a detecting of the portion of text on every frame using the OCR operation consumes more computational resources than applying the interframe image transformation.

[0136] In some embodiments, a detecting of the portion of text on every frame using the OCR operation causes a higher latency in a higher latency in the providing of the tracked portion of text.

[0137] Some embodiments involve, responsive to the receiving of the user query, providing a notification that the search for the portion of text is being performed.

[0138] Some embodiments involve providing a notification that the portion of text has been detected in the live preview.

[0139] Some embodiments involve, responsive to the initiating of the tracking of the detected portion of text, providing a notification that the tracking of the detected portion of text is being performed.

[0140] In some embodiments, the notification comprises one or more of an audio notification, a haptic notification, or a visual notification.

[0141] In some embodiments, the tracked portion of text comprises a plurality of words or phrases.

[0142] Some embodiments involve, prior to the detecting of whether the portion of text in the received user query is present in the first frame, determining whether the first frame comprises textual content. Such embodiments involve, upon a determination that the first frame does not comprise textual content, not performing the OCR operation on the first frame.

[0143] In some embodiments, the portion of text may be a character or a plurality of characters within the larger portion of text.

[0144] In some embodiments, the computing device may be a mobile device.

[0145] Figure 11 is a flowchart of a method, in accordance with example embodiments. Method 1100 may include various blocks or steps. The blocks or steps may be carried out individually or in combination. The blocks or steps may be carried out in any order and / or in series or in parallel. Further, blocks or steps may be omitted or added to method 1100.

[0146] The blocks of method 1100 may be carried out by various elements of computing device 900 as illustrated and described in reference to Figure 9.

[0147] Block 1110 involves receiving a user query to search for a target in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames.

[0148] Block 1120 involves detecting, by the image capturing device, whether the target is present in a first frame of the plurality of frames, wherein the detecting comprises applying a detection algorithm on at least a larger context of the scene in the first frame.

[0149] Block 1130 involves, responsive to detecting that the target is present in the first frame: pausing the detection algorithm on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected target in the larger context of the scene, wherein the predicting is based on a previously determined location of the detected target relative to the larger context, and initiating a tracking of the detected target in the frames successive to the first frame based on the predicted relative location of the detected target, wherein the tracking of the detected target is based on tracking the larger context and the previously determined location of the detected target relative to the larger context.

[0150] Block 1140 involves providing, in the live preview comprising the frames successive to the first frame, the tracked target.

[0151] In some embodiments, the target may be a character or a plurality of characters, and the larger context may be a larger portion of text.

[0152] In some embodiments, the detection algorithm may be an OCR operation.

[0153] In some embodiments, the target may be an object within the larger context, and wherein the detection algorithm may be an object detection algorithm. Some embodiments involve tracking the larger context using a tracking algorithm that is different from the object detection algorithm. In some embodiments, the object detection algorithmrequires greater computing resources than the tracking algorithm.

[0154] In some embodiments, the larger context is a large diagram, and the object is a smaller portion of the large diagram.

[0155] In some embodiments, the predicting may be based on an inter-frame image transformation of the larger context between the first frame and a second frame successive to the first frame. In some embodiments, the inter-frame image transformation may be a homography matrix.

[0156] In some embodiments, the receiving of the user query is performed in a multimodal manner. In some embodiments, the multimodal manner includes one or more of an image modality, a text modality, or a voice modality.

[0157] In some embodiments, the receiving of the user query is performed in a multilingual manner.

[0158] In some embodiments, a detecting of the target on every frame using the detection algorithm causes a higher latency in the providing of the tracked target.

[0159] In some embodiments, the target may be located outside the larger context. For example, the target may be a legend of a drawing (the larger context), and the legend may be located outside the drawing. As another example, the target may be a traffic symbol, and the larger context may be a large building near the traffic symbol.

[0160] The particular arrangements shown in the Figures should not be viewed as limiting. It should be understood that other embodiments may include more or less of each element shown in a given Figure. Further, some of the illustrated elements may be combined or omitted. Yet further, an illustrative embodiment may include elements that are not illustrated in the Figures.

[0161] A step or block that represents a processing of information can correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a step or block that represents a processing of information can correspond to a module, a segment, or a portion of program code (including related data). The program code can include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and / or related data can be stored on any type of computer readable medium such as a storage device including a disk, hard drive, or other storage medium.

[0162] The computer readable medium can also include non-transitory computer readable media such as computer-readable media that store data for short periods of timelike register memory, processor cache, and random access memory (RAM). The computer readable media can also include non-transitory computer readable media that store program code and / or data for longer periods. Thus, the computer readable media may include secondary or persistent long-term storage, like read only memory (ROM), optical or magnetic disks, compact disc read only memory (CD-ROM), for example. The computer readable media can also be any other volatile or non-volatile storage systems. A computer readable medium can be considered a computer readable storage medium, for example, or a tangible storage device.

[0163] While various examples and embodiments have been disclosed, other examples and embodiments will be apparent to those skilled in the art. The various disclosed examples and embodiments are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

CLAIMSWe claim:

1. A computer-implemented method comprising:receiving a user query to search for a portion of text in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames;detecting, by the image capturing device, whether the portion of text in the received user query is present in a first frame of the plurality of frames, wherein the detecting comprises performing an optical character recognition (OCR) operation on at least a larger portion of text in the first frame;responsive to detecting that the portion of text is present in the first frame:pausing the OCR operation on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of the detected portion of text in the larger portion of text, wherein the predicting is based on a previously determined location of the detected portion of text in the larger portion of text in the first frame, andinitiating a tracking of the detected portion of text in the frames successive to the first frame based on the predicted relative location of the detected portion of text, wherein the tracking of the detected portion of text is based on tracking the larger portion of text and the previously determined location of the detected portion of text within the larger portion of text; andproviding, in the live preview comprising the frames successive to the first frame, the tracked portion of text.

2. The computer-implemented method of claim 1, further comprising: responsive to the receiving of the user query, decreasing a frame rate for the image capturing device.

3. The computer-implemented method of claim 1, further comprising: responsive to the receiving of the user query, lowering an image resolution for the plurality of frames.

4. The computer-implemented method of claim 1, further comprising:responsive to the receiving of the user query, converting the plurality of frames to grayscale images.

5. The computer-implemented method of claim 1, wherein the larger portion of text comprises a number of words that exceeds a threshold number.

6. The computer-implemented method of claim 1, wherein the pausing of the OCR operation comprises pausing the OCR operation until a next application of the OCR operation at a next frame in the plurality of frames, and further comprising:applying the OCR operation at the next frame.

7. The computer-implemented method of claim 6, wherein the next application of the OCR operation is based on a rate of application of the OCR operation.

8. The computer-implemented method of claim 7, wherein the rate of application of the OCR operation is based on an analysis of the scene.

9. The computer-implemented method of claim 1, wherein the receiving of the user query comprises receiving a textual input.

10. The computer-implemented method of claim 1, wherein the receiving of the user query comprises receiving a voice input.

11. The computer-implemented method of claim 1, wherein the receiving of the user query comprises receiving the user query in a language other than English.

12. The computer-implemented method of claim 1, wherein the larger portion of text is in a language other than English.

13. The computer-implemented method of claim 1, further comprising: detecting that the portion of text is not present in the first frame; andnot performing the OCR operation on the first frame.

14. The computer-implemented method of claim 13, further comprising:performing the OCR operation on another larger portion of text in another frame of the plurality of frames.

15. The computer-implemented method of claim 1 , wherein the providing of the tracked portion of text comprises displaying an enlarged view of a portion of the live preview that comprises the tracked portion of text.

16. The computer-implemented method of claim 1 , wherein the providing of the tracked portion of text comprises highlighting the tracked portion of text.

17. The computer-implemented method of claim 1 , wherein the providing of the tracked portion of text comprises reducing a screen brightness level for a portion of the live preview outside the tracked portion of text.

18. The computer-implemented method of claim 1, further comprising: determining an inter-frame image transformation of the larger portion of text between the first frame and a second frame successive to the first frame, and wherein the predicting is based on the inter-frame image transformation.

19. The computer-implemented method of claim 18, wherein the inter-frame image transformation is a homography matrix.

20. The computer-implemented method of claim 18, wherein a detecting of the portion of text on every frame using the OCR operation consumes more computational resources than applying the inter-frame image transformation.

21. The computer-implemented method of claim 18, wherein a detecting of the portion of text on every frame using the OCR operation causes a higher latency in a higher latency in the providing of the tracked portion of text.

22. The computer-implemented method of claim 1, further comprising: responsive to the receiving of the user query, providing a notification that the search for the portion of text is being performed.

23. The computer-implemented method of claim 1, further comprising: providing a notification that the portion of text has been detected in the live preview.

24. The computer-implemented method of claim 1, further comprising: responsive to the initiating of the tracking of the detected portion of text, providing a notification that the tracking of the detected portion of text is being performed.

25. The computer-implemented method of any of claims 20-22, wherein the notification comprises one or more of an audio notification, a haptic notification, or a visual notification.

26. The computer-implemented method of claim 1, wherein the tracked portion of text comprises a plurality of words or phrases.

27. The computer-implemented method of claim 1, further comprising: prior to the detecting of whether the portion of text in the received user query is present in the first frame, determining whether the first frame comprises textual content; andupon a determination that the first frame does not comprise textual content, not performing the OCR operation on the first frame.

28. The computer-implemented method of claim 1, wherein the portion of text is a character or a plurality of characters.

29. A computer-implemented method comprising:receiving a user query to search for a target in a live preview of a scene being captured by an image capturing device, wherein the live preview comprises a plurality of frames;detecting, by the image capturing device, whether the target is present in a first frame of the plurality of frames, wherein the detecting comprises applying a detection algorithm on at least a larger context of the scene in the first frame;responsive to detecting that the target is present in the first frame:pausing the detection algorithm on frames successive to the first frame, predicting, in frames successive to the first frame, a relative location of thedetected target in the larger context of the scene, wherein the predicting is based on a previously determined location of the detected target relative to the larger context, andinitiating a tracking of the detected target in the frames successive to the first frame based on the predicted relative location of the detected target, wherein the tracking of the detected target is based on tracking the larger context and the previously determined location of the detected target relative to the larger context; andproviding, in the live preview comprising the frames successive to the first frame, the tracked target.

30. The computer-implemented method of claim 29, wherein the target is a character or a plurality of characters, and the larger context is a larger portion of text.

31. The computer-implemented method of claim 30, wherein the detection algorithm comprises an OCR operation.

32. The computer-implemented method of claim 29, wherein the target is an object within the larger context, and wherein the detection algorithm comprises an object detection algorithm.

33. The computer-implemented method of claim 32, further comprising tracking the larger context using a tracking algorithm that is different from the object detection algorithm.

34. The computer-implemented method of claim 33, wherein the object detection algorithm requires greater computing resources than the tracking algorithm.

35. The computer-implemented method of claim 29, wherein the larger context is a large diagram, and the object is a smaller portion of the large diagram.

36. The computer-implemented method of claim 29, wherein the predicting is based on an inter-frame image transformation of the larger context between the first frame and a second frame successive to the first frame.

37. The computer-implemented method of claim 36, wherein the inter-frame image transformation is a homography matrix.

38. The computer-implemented method of claim 29, wherein the receiving of the user query is performed in a multimodal manner.

39. The computer-implemented method of claim 38, wherein the multimodal manner comprises one or more of an image modality, a text modality, or a voice modality.

40. The computer-implemented method of claim 29, wherein the receiving of the user query is performed in a multilingual manner.

41. The computer-implemented method of claim 29, wherein a detecting of the target on every frame using the detection algorithm causes a higher latency in the providing of the tracked target.

42. The computer-implemented method of claim 29, wherein the target is located outside the larger context.

43. A computing device, comprising:one or more processors; anddata storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out functions that comprise the computer-implemented method of any one of claims 1-42.

44. The computing device of claim 43, wherein the computing device is a mobile device.

45. An article of manufacture comprising one or more non-transitory computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to carry out functions that comprise the computer-implemented method of any one of claims 1-42.

46. A program that, when executed by one or more processors of a computing device, causes the computing device to carry out functions that comprise the computer-implemented method of any one of claims 1-42.

Images