Systems, methods, and media for switching visual attention models

Abstract

A method includes receiving an input content representing a visual scene and processing the input content to obtain available information related to the visual scene. The method further includes determining a visual attention model (VAM) based on the available information and generating predicted visual saliency for regions of the visual scene using the determined VAM.

Description

PA101827W002SYSTEMS, METHODS, AND MEDIA FOR SWITCHING VISUAL ATTENTION MODELSTechnical Field

[0001] The present disclosure relates to a non-transitory computer-readable medium for switching visual attention models.Background

[0002] Eye tracking has been used primarily for marketing or design testing purposes. Uses of eye tracking have included human computer interaction (e.g., for design and for user assistance), usability testing, and research on hard goods. Other uses of eye tracking include market research and advertising (e.g., which elements, messages, or designs capture user’s attention and subsequently match the user’s goals), medical and clinical diagnosis, assessing fatigue or alertness, and / or education and learning (e.g., can be used to detect reading difficulties). However, eye tracking requires the user to be near or even wear eye-tracking devices that are often expensive and / or bulky. This has led to individuals utilizing visual attention modeling as a proxy for eye-tracking data. Additionally, eye tracking is passive and does not inform the user or others what is impacting visual attention and therefore cannot inform them how they might adjust the visual scene so as to alter visual saliency. Further, eye tracking does not predict future saccades such that this information might be used to provide feedback (e.g., highlighting predicted unattended regions of the scenes, etc.).

[0003] Therefore, there is a need for an improved solution for addressing the abovementioned shortcomings of the existing visual attention analysis tools.Summary

[0004] In a first aspect, at least one non-transitory computer-readable medium storing instructions that, when executed, configure at least one processor to receive an input content representing a visual scene and process the input content to obtain available information related to the visual scene. The method further includes instructions to determine a visual attention model (VAM) based on the available information and generate predicted visual saliency for regions of the visual scene using the determined VAM.

[0005] In a second aspect, at least one system includes at least one computing device that includes one or more processors. At least one memory coupled to at least one of the one or more processors, wherein the at least one memory includes instructions that configure the at least one computing device to receive an input content representing a visual scene and processthe input content to obtain available information related to the visual scene. The method further includes instructions to determine a visual attention model (VAM) based on the available information and generate predicted visual saliency for regions of the visual scene using the determined VAM.

[0006] In a third aspect, a computer-implemented method includes receiving an input content representing a visual scene and processing the input content to obtain available information related to the visual scene. The method further includes determining a visual attention model (VAM) based on the available information and generating predicted visual saliency for regions of the visual scene using the determined VAM.

[0007] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims.Brief Description of the Drawings

[0008] Exemplary embodiments disclosed herein may be more completely understood in consideration of the following detailed description in connection with the following figures. The figures are not necessarily drawn to scale. Like numbers used in the figures refer to like components. However, it will be understood that the use of a number to refer to a component in a given figure is not intended to limit the component in another figure labeled with the same number.

[0009] FIG. 1 shows a schematic block diagram of at least one system for generating predicted visual saliency for regions of a visual scene, according to an embodiment of the present disclosure;

[0010] FIG. 2 shows a schematic block diagram of at least one processor of the at least one system, according to an embodiment of the present disclosure;

[0011] FIG. 3 shows a flowchart of a method for generating predicted visual saliency for regions of a visual scene, according to an embodiment of the present disclosure;

[0012] FIG. 4 shows a flowchart of an exemplary process for VAM analysis, according to an embodiment of the present disclosure;

[0013] FIG. 5 illustrates an exemplary embodiment of a segmented entity that is constrained to specific hue values, according to an embodiment of the present disclosure;

[0014] FIGS. 6A and 6B illustrate exemplary embodiments where a physical product is distinguished by specific color values based on production years, according to an embodiment of the present disclosure;

[0015] FIG. 7 illustrates an exemplary user interface displaying application of different modification constraints and visual attention goals to multiple areas of interest, according to an embodiment of the present disclosure;

[0016] FIG. 8 illustrates an exemplary user interface displaying a user-defined gaze sequence goal that spans multiple areas of interest, according to an embodiment of the present disclosure;

[0017] FIG. 9 illustrates an exemplary user interface displaying a user-defined gaze sequence goal that spans multiple areas of interest without a specified order, according to an embodiment of the present disclosure; and

[0018] FIGS. 10 A- 10C illustrate exemplary user interfaces describing generation of an attention-weighted image from an original scene, according to an embodiment of the present disclosure.Detailed Description

[0019] In the following description, reference is made to the accompanying figures that form a part thereof and in which various embodiments are shown by way of illustration. It is to be understood that other embodiments are contemplated and may be made without departing from the scope or spirit of the present disclosure. The following detailed description, therefore, is not to be taken in a limiting sense.

[0020] In the following disclosure, the following definitions are adopted.

[0021] As used herein, all numbers should be considered modified by the term “about”. As used herein, “a,” “an,” “the,” “at least one,” and “one or more” are used interchangeably.

[0022] As used herein as a modifier to a property or attribute, the term “generally”, unless otherwise specifically defined, means that the property or attribute would be readily recognizable by a person of ordinary skill but without requiring absolute precision or a perfect match (e.g., within + / - 20 % for quantifiable properties).

[0023] The term “substantially”, unless otherwise specifically defined, means to a high degree of approximation (e.g., within + / - 10% for quantifiable properties) but again without requiring absolute precision or a perfect match.

[0024] The term “about”, unless otherwise specifically defined, means to a high degree of approximation (e.g., within + / - 5% for quantifiable properties) but again without requiring absolute precision or a perfect match.

[0025] As used herein, the terms “first” and “second” are used as identifiers. Therefore, such terms should not be construed as limiting of this disclosure. The terms “first” and “second”when used in conjunction with a feature or an element can be interchanged throughout the embodiments of this disclosure.

[0026] As used herein, “at least one of A and B” should be understood to mean “only A, only B, or both A and B”.

[0027] As used herein, the term “user” generally refers to a person who wishes to enhance or modify a scene or an image. The user may include an owner of a product or a service provider.

[0028] As used in some embodiments herein, the term “entity” generally refers to a thing or an area or a region within a scene, or regions within a scene that may be analyzed.

[0029] As used in some embodiments herein, the term “saliency metric” generally refers to an amount of attention attracted at a time of viewing a scene or an image.

[0030] As used in some embodiments herein, the term “attention score” generally refers to an amount of attention given to a specific area in an image or a scene.

[0031] As used in some embodiments herein, the term “gaze direction” generally refers to a direction in which a viewer is likely to view a region of a visual representation.

[0032] As used in some embodiments herein, the term “gaze sequence” generally refers to an order or a sequence in which a viewer is likely to view different regions of a visual representation.

[0033] Prediction of visual attention in a visual scene, such as determining which parts of the visual scene are most salient, looked at, or attended to, and a likely order in which those elements will be viewed, is a component in various fields of analysis and design. For example, such analysis is applicable to merchandise placement, environmental design, safety assessments, web page layout, advertising effectiveness, and product marketing. Designers may want to identify components in an advertisement or design that they would like to get more attention (e.g., brand logos, product image, and key message items). Architects may want to design a space to draw attention to certain elements in the area, like an exit in an office building or an island in a kitchen.

[0034] Visual attention modeling seeks to predict which entities in an image or scene that a user’s attention will be drawn to, and the order in which the user will attend to those entities. However, its predictions may be less accurate in circumstances where the user has a specific visual goal (such as they are looking for a particular item or product) or in situations with unusual visual stimulus (e.g., objects that are out of context with the rest of the scene). There are some visual attention models that stimulate how the human brain selectively attends specific regions or features of a visual scene based on higher-level cognitive processes and goals. However, these visual attention models may have limited scope as they are typicallytrained on very specific tasks or applications. Therefore, such visual attention models may fail to be able to generalize to diverse data sets and applications. Therefore, there is a need for an improved solution for addressing the abovementioned shortcomings of the existing visual attention analysis tools.

[0035] According to aspects of the disclosure, at least one non-transitory computer-readable medium stores instructions that, when executed, configure at least one processor to: receive an input content representing a visual scene; process the input content to obtain available information related to the visual scene; determine a visual attention model (VAM) based on the available information; and generate predicted visual saliency for regions of the visual scene using the determined VAM.

[0036] Some uses of the VAM may include object recognition (for example by selectively attending to informative regions, computer vision (CV) can improve object classification), image captioning (such as making sure that the most salient objects in the visual scene are captured in the descriptions), and / or image generation (informing GenAI where resources should be concentrated, i.e., which parts are most important to get right). Other uses may include action recognition, visual tracking, marketing and advertising (predicting which elements, messages, or designs capture a viewer’s attention and subsequently matches the user’s goals), and / or education and training (such as predicting which elements will be attended to and which will be overlooked, can compare to ideal or expert users). Some of these capabilities are built into the VAM or multiple VAMs itself, while others, like object recognition and image generation, are provided by other models or technologies that are integrated with a VAM.

[0037] The at least one non-transitory computer-readable medium of the present disclosure may intelligently switch between the VAMs, such as bottom up, top down, and hybrid VAMs to predict visual attention and evaluate the saliency for the regions of the visual scene, such as an image, a video, a design, etc. The predicted visual attention and the evaluated saliency for the regions of the visual scene may be more accurate across a wide range of visual scenes, media types, and user states. This is because the at least one non-transitory computer-readable medium, may determine when to use a particular VAM for a given application. Thus, the at least one non-transitory computer-readable medium may ensure an appropriate VAM is deployed for a given context. Furthermore, this intelligent switching or dynamic determination may allow the at least one non-transitory computer-readable medium to allocate computational resources by choosing a less-intensive VAM when a complex goal-inference is unnecessary, thereby improving overall efficiency.

[0038] Referring now to figures, FIG. 1 shows a schematic block diagram of at least one system 100 for generating predicted visual saliency for regions of a visual scene, according to an embodiment of the present disclosure.

[0039] The at least one system 100 is configured to receive an input content representing the visual scene. The at least one system 100 is further configured to process the input content to obtain available information related to the visual scene. The at least one system 100 is further configured to determine a visual attention model (VAM) based on the available information. The at least one system 100 is further configured to generate the predicted visual saliency and gaze sequence for the regions of the visual scene using the determined VAM.

[0040] The at least one system 100 includes at least one computing device 101 (interchangeably referred to as “the computing device 101” herein), through which embodiments of the present disclosure may be implemented, such as those depicted and / or described in other figures herein. The computing device 101 described herein is but one example of a suitable computing device and does not suggest any limitation on the scope of any embodiments presented. Nothing illustrated or described with respect to the computing device 101 should be interpreted as being required or as creating any type of dependency with respect to any element or plurality of elements. In some embodiments, the computing device 101 may include, but need not be limited to, a desktop, laptop, server, client, tablet, smartphone, computing cloud or any other type of device that can utilize data.

[0041] In the illustrated embodiment of FIG. 1, the computing device 101 includes at least one processor 102 and memory 110 including a non-volatile memory 108, a volatile memory 108’, and / or a combination thereof. The term “at least one processor 102” is interchangeably referred to herein as “the one or more processors 102”. Specifically, the at least one computing device 101 includes the one or more processors 102. The at least one memory 110 is coupled to at least one of the one or more processors 102.

[0042] The computing device 101 may include one or more displays, display hardware, and / or output devices 104 such as, for example, Augmented Reality / Virtual Reality / Mixed Reality / Extended Reality hardware, monitors, speakers, headphones, projectors, wearabledisplays, holographic displays, and / or printers. The output devices 104 may further include, for example, displays and / or speakers, devices that emit energy (radio, microwave, infrared, visible light, ultraviolet, x-ray and gamma ray), electronic output devices (Wi-Fi, radar, laser, etc.), audio (of any frequency), and the like.

[0043] The computing device 101 may further include one or more input devices 106 which can include, by way of example, any type of mouse, keyboard, disk / media drive, memory stick / thumb-drive, memory card, pen, touch-input device, biometric scanner, gazeand / or blink tracker, tracker, voice / auditory input device, motion-detector, camera, scale, and any device capable of measuring data such as motion data (e.g., an accelerometer, Global Positioning System (GPS), a magnetometer, a gyroscope, etc.), biometric data (e.g., blood pressure, pulse, heart rate, perspiration, temperature, voice, facial-recognition, motion / gesture tracking, gaze tracking, iris or other types of eye recognition, hand geometry, oxygen saturation, glucose level, fingerprint, DNA, dental records, weight, or any other suitable type of biometric data, etc.), videos, still images, and audios (including human-audible and human-inaudible ultrasonic sound waves). The one or more input devices 106 may include any type of device capable of receiving data, whether from another device, visual and / or audio data captured from a real world, object detection data, and the like. The one or more input devices 106 may include cameras (with or without audio recording), such as digital and / or analog cameras, still cameras, video cameras, thermal imaging cameras, infrared cameras, cameras with a charge-couple display, night-vision cameras, three-dimensional cameras, webcams, audio recorders, and the like.

[0044] The computing device 101 typically includes the non-volatile memory 108 (e.g., Read-only memory (ROM), flash memory, etc.), the volatile memory 108’ (e.g., Randomaccess memory (RAM), etc.), or a combination thereof.

[0045] As shown in FIG. 1, the computing device 101 further includes a network interface 112 to facilitate communications over a network 114 with other data source such as a database 118 via wires, a wide area network, a local area network, a personal area network, a cellular network, a satellite network, and the like. The suitable local area networks may include wired Ethernet and / or wireless technologies such as, for example, wireless fidelity (Wi-Fi). The suitable personal area networks may include wireless technologies such as, for example, Infrared Data Association (IrDA), Bluetooth, Wireless USB, Z-Wave, ZigBee, and / or other near field communication protocols. The suitable personal area networks may similarly include wired computer buses such as, for example, USB and FireWire. The suitable cellular networks may include, but are not limited to, technologies such as Long Term Evolution (LTE), Worldwide Interoperability for Microwave Access (WiMAX), Universal Mobile Telecommunications System (UMTS), Code Division Multiple Access (CDMA), and Global System for Mobile Communications (GSM). The network interface 112 may be communicatively coupled to any device capable of transmitting and / or receiving data via one or more network(s) (e.g., the network 114). Accordingly, the network interface 112 may include a communication transceiver for sending and / or receiving any wired or wireless communication. For example, the network interface 112 may include an antenna, a modem, LAN port, Wi-Fi card, WiMax card, mobile communications hardware, near-fieldcommunication hardware, satellite communication hardware and / or any wired or wireless hardware for communicating with other networks and / or devices.

[0046] As shown in FIG. 1, the computing device 101 further includes a computer-readable medium 116 including one or more plurality of computer-readable media, each of which is non-transitory. The computer-readable medium 116 may reside, for example, within the input device 106, the non-volatile memory 108, the volatile memory 108’, or any combination thereof. The at least one memory 110 may include tangible media that is able to store instructions associated with, or used by, the computing device 101 or the system 100. The computer-readable medium 116, also referred to herein as a non-transitory computer-readable medium, includes, by way of non-limiting examples: RAM, ROM, cache, fiber optics, Erasable Programmable Read-Only Memory (EPROM) / Flash memory, Compact Disc (CD) / Digital versatile disc (DVD) / Blu-ray Disc Read-Only Memory (BD-ROM), hard disk drives, solid-state storage, optical or magnetic storage devices, diskettes, electrical connections having a wire, or any combination thereof. The non-transitory computer-readable medium 116 may also include, for example, a system or device that is of a magnetic, optical, semiconductor, or electronic type. The non-transitory computer-readable medium 116 excludes carrier waves and / or propagated signals taking any number of forms such as optical, electromagnetic, or combinations thereof.

[0047] The computing device 101 may include the one or more network interfaces 112 to facilitate communication with one or more remote devices, which may include, for example, client and / or server devices. The network interface 112 may also be described as a communications module, as these terms may be used interchangeably. The database 118 is depicted as being accessible over the network 114 and may reside within a server, a cloud, or any other configuration to support being able to remotely access data and store data in the database 118.

[0048] FIG. 2 shows a schematic block diagram of the at least one processor 102 of the at least one system 100 shown in FIG. 1, according to an embodiment of the present disclosure.

[0049] Referring to FIGS. 1 and 2, the at least one processor 102 is configured to receive the input content representing the visual scene. In various embodiments, the at least one processor 102 receives the input content from the one or more input devices 106 and presents an output on the one or more output devices 104.

[0050] As shown in FIG. 2, the at least one processor 102 includes a plurality of VAMs 204a, 204b, 204c for generating a VAM output. The plurality of VAMs 204a, 204b, 204c may be individually or collectively referred to as “the VAM 204” herein. The VAM output may include predicted visual saliency, an attention score, or a map of the visual scene. The VAM204 may analyze the visual scene to identify one or more regions including user-defined AOIs, distractor regions, and non-salient regions. The VAM may analyze the entire scene to produce an attention heatmap or saliency map. Based on this map, user inputs defining AOIs, and predefined criteria, the system 100 partitions the scene into the one or more regions for a holistic analysis.

[0051] In some embodiments, the at least one processor 102 includes one or more segmentation models 202 for identifying pixels associated with entities in an image or a video frame, for example, within a user-defined AOI or other identified regions of the visual scene. Specifically, the one or more segmentation models 202 may analyze the one or more regions and identify objects or entities therein.

[0052] In some embodiments, the at least one processor 102 further includes one or more optimization models 206 for modifying the segmented entities within the one or more regions to achieve user goals, which may include optimizing the VAM output. A user goal refers to a user-desired manner in which a user or a viewer attends to the entities within the visual scene. For example, the user goals may relate to specific objects that are attended to in addition to collection of objects that are deemed unimportant or even detrimental, specific objects attended to in a particular sequence or at a particular time, or specific objects attended to from a particular viewing point, and the like.

[0053] Specifically, the one or more optimization models 206 may be configured to modify the segmented entities within any of the identified regions (e.g., AOIs, distractors, or NSRs) to achieve the user goals. In some embodiments, the at least one processor 102 further includes a classification model 208 for classifying each segmented entity to assign a class type (e.g., a text, an object, etc.). In some embodiments, the VAM 204, the one or more optimization model 206, the one or more segmentation models 202, and the classification model 208 may function iteratively to enhance the visual scene according to the user goals.

[0054] FIG. 3 shows a flowchart of a method 200 for generating predicted visual saliency for regions of a visual scene, according to an embodiment of the present disclosure. Note that the method 200 is configured to be performed using the at least one system 100 shown in FIG.1. Specifically, the method 200 is configured to be performed by the at least one computing device 101 of the at least one system 100. More specifically, the at least one non-transitory computer-readable medium 116 is configured to store instructions that, when executed, configure the at least one processor 102 to perform operations of the method 200. The method 200 is described with additional reference to FIGS. 1 and 2.

[0055] At operation 210, the method 200 includes receiving an input content representing the visual scene. The input content may be received by the at least one computing device 101 using the one or more input devices 106 or from the database 118 over the network 114.

[0056] At operation 212, the method 200 further includes processing the input content to obtain available information related to the visual scene. In some embodiments, processing the input content includes performing one or more image analytics operations including object detection, segmentation, feature extraction, or object tracking. This may transform a raw visual scene into a structured and machine-readable format including identified objects and their associated features.

[0057] In some embodiments, the available information includes explicit user inputs including voice input, text input, graphical interface input, or search terms. The user inputs may directly ascertain the user’ s specific goal or task. This may eliminate ambiguity and a need for inference, allowing for a highly accurate and rapid determination of the VAM, which may further improve a relevance and correctness of the saliency prediction.

[0058] In some embodiments, the available information includes behavioral indicators including eye movement patterns, gaze directi on / sequence, real world interactions, or movement patterns. This may enable inferring the user intent by observing the user’s implicit actions and behaviors. This may allow determination of the VAM even when the user does not explicitly state their goal or task and provide a non-intrusive method for detecting the goals or task, thereby enhancing saliency prediction accuracy.

[0059] For example, if a user looks at upper right corner of a webpage, they may be looking to close browser. In another example, the eye movement patterns and gaze direction may include looking at semantically related items. In another example, the real world interactions may include picking up semantically related items. In some examples, the movement patterns may include stopping patterns, what objects they look at, pick-up, put in cart, etc. (e.g., store path, pausing at or near certain product, looking at eye fixation data).

[0060] At operation 214, the method 200 further includes determining a visual attention model (VAM) (e.g., the VAM 204) based on the available information. Specifically, at the operation 214, the method 200 further includes determining the VAM from the plurality of VAMs based on the available information. This understanding of the visual scene’s components from the operation 212 may provide a set of data points, enabling a more granular, informed, and accurate selection of the appropriate VAM.

[0061] In some embodiments, determining the VAM includes selecting between a bottom up model (or a bottom up VAM), a top down model (or a top down VAM), or a hybrid model (or a hybrid VAM) based on whether the user is determined to have a specific task or goal.This may establish a clear and logical criterion for the VAM selection process. This may optimize both accuracy and computational efficiency, i.e., a computationally efficient bottom-up model may be selected for passive viewing scenarios while a more sophisticated and accurate top-down or hybrid model may be selected only when a goal-oriented task is detected.

[0062] In some embodiments, determining the VAM includes applying collaborative filtering based on behavioral patterns of users having similar profiles or actions. For example, the collaborative filtering may leverage behavior and preferences of similar users to infer intent, such as similar shoppers also purchased X. In some embodiments, determining whether the user has a specific task or goal includes analyzing contextual data including to do lists, calendar events, or user profiles. Therefore, the awareness may be expanded beyond the visual scene to encompass the user’ s broader work or life context. This may enable proactively and accurately inferring the user intent, for example by anticipating a goal-oriented search based on an upcoming calendar event, thereby allowing for a preemptive and correct configuration of the VAM.

[0063] In one embodiment, the bottom up VAM is a stimulus-driven process and occurs involuntarily in response to physical features of a visual scene. The bottom up model is guided by salient or attention grabbing features such as bright colors, high saturation, high contrast, or sudden movements or flashes. The bottom up VAM may do a good job of predicting where an average user is going to look in the first 3-5 seconds of the visual scene, but can lack the ability to factor in user goals, scene understanding, temporal modeling, etc. to predict visual attention. For example, if the user or the viewer is actively searching for their keys in a room, their eye fixations would likely be different than if they were simply passively viewing the same room. Similarly, saliency of a particular object might be affected by the rest of the visual scene (or scene understanding). For example, a toaster on a kitchen counter is likely less likely to capture the viewers’ attention than the same toaster if placed on a bathroom counter.

[0064] In one embodiment, the top down VAM is a process driven by cognitive factors, such as user goals, expectations, and prior knowledge. The top down model integrates the user goals, expectations, prior knowledge, image analytics and scene understanding, as well as the bottom up VAM when predicting visual attention. The top down VAMs of visual attention (i.e., computational models that stimulate how a human brain selectively attends specific regions or features of the visual scene based on higher-level cognitive processes and goals). The top down VAMs may have limited scope due to typically being trained on very specific tasks or applications and may fail to be able to generalize to diverse data sets and applications. Embodiments may combine components from various top down VAMs. For example, a top down model of visual attention could have both feature-based and space-based selectionmechanisms. Space-based selection mechanisms would likely be useful for: dynamic stimuli, attentional blink phenomenon, inhibition of return, central bias effect, modeling effect of history previous visual attention locations, etc. Visual attention could be influenced by features of the object and scene and preferential weights could be given to whole objects rather than individual features (e.g., sharing features in common with the object you are actively looking for might increase the likelihood of attending to that region of the scene but sharing multiple or all features in common to the searched for item without any mismatched features could provide bonus saliency weightings). In some embodiments, visual attention could be driven by competitive comparison between number and level of feature matches between different regions in the visual scene and how they match with current search features. The region with the highest match would win attention. For example: if the user is searching for a TV remote (i.e., rectangular device, approximately 1.5” x 6” x 0.33” in size, plastic, multiple buttons, etc.), a keyboard might have three features in common with the search query (plastic and multiple buttons), and a calculator might match all four features with the search query. Therefore, according to the top down model, the calculator would win in capturing attention (if all other things were equal).

[0065] In one embodiment, the hybrid VAMs combine one or more bottom up VAMs anchored in context independent features like color contrasts and oriented edges with one or more top down VAMs, which are driven by context and intent dependent features like object and language recognition and scene anomaly detection, like a fire hydrant inside an indoor office. Commercially available bottom up VAMs estimate visual attention based on a presence and density of features like color contrasts, oriented edges and color intensity in a scene, such as a still image or a frame of a video. These bottom up VAMs may provide no way for a user to codify intent or context, such as the user is searching for something specific, nor do they recognize that certain objects do not belong together, like a fire hydrant inside an indoor office. These and other gaps in capabilities of the bottom up VAMs can be filled by fusing the top down VAMs with the bottom up VAMs. As discussed above, the top down VAMs estimate attention drawn to different areas in a visual scene, like a still image or a frame of a video, based on entities that align with the user goals or contextual awareness. For example, if the user has a goal of finding food and peers into a restaurant, their attention may be drawn to a menu or cooked food being placed on or carried on dishware more than other areas in the visual scene. The top down VAMs can make such areas of the visual scene more salient given an input goal(s) of the user and understanding of anomalous content elements. Combining the top down and bottom up VAMs into the hybrid VAM may create a more comprehensive system that can accommodate more inputs from the user.

[0066] At operation 216, the method 200 further includes generating the predicted visual saliency for the regions of the visual scene using the determined VAM (e.g., the top down VAM, the bottom up VAM, or the hybrid VAM). In some embodiments, generating the predicted visual saliency includes weighing whole objects based on similarity between object features and features associated with an inferred user goal. The saliency calculation may therefore be at a cognitively relevant object level, rather than at a disconnected pixel or feature level. The resulting saliency prediction may be more closely aligned with human perceptual processes, resulting in a more realistic and useful predicted saliency.

[0067] In some embodiments, the method 200 further includes adjusting parameters of the determined VAM based on semantic relationships among scene components or based on matches between scene components and features associated with the inferred user goal. The determined VAM may not be merely chosen, but may also be specifically fine-tuned and configured for particularities of the visual scene and the user's goal. This may provide a more granular level of adaptation, where saliency weights for specific features or objects consistent with the user’s goal may be increased, leading to a more refined and accurate final saliency prediction.

[0068] In some embodiments, the method 200 further includes training one or more of the visual attention models using visual attention data obtained from eye-tracking devices in combination with task related contextual information. The creation of this closed-loop learning system may be iteratively improved using ground-truth data, e.g., the visual attention data obtained from the eye-tracking devices. Thus, the VAMs and its model-switching logic may progressively be more accurate and effective over time, as it refines based on real-world correlations between visual attention, context, and tasks, thereby enhancing the long-term performance and robustness.

[0069] In some embodiments, the visual attention data may be obtained from direct measures, such as eye-tracking data from the eye-tracking devices. The eye-tracking devices may include augmented reality devices. The visual attention data may be used to train and therefore improve the VAMs and when to switch between the different VAMs. This may include improving both the bottom up and top down VAMs via the visual attention data as training data and the visual scene (i.e., ideally captured from a camera also in the augmented reality devices) in concert with knowledge about a task of the user. For example, the knowledge may include that the user has a specific task that they are doing. For example, the user may have a certain shopping list they are looking for or they are just browsing. In some embodiments, the visual attention data may be obtained from indirect measures, such ascomparison of model’ s predictions and less direct user actions that might not be directly related to visual attention but might be related to obtaining the user’s goals.

[0070] FIG. 4 shows a flowchart of a process 250 for VAM analysis, according to an embodiment of the present disclosure. The process 250 provides a detailed view of the operations described in FIG. 3. The process 250 is embodied as one or more algorithms implemented by the at least one processor 102 (shown in FIG. 1) of the at least one computing device 101 (shown in FIG. 1). Further, the process 250 may be stored in the at least one memory 110 (shown in FIGS. 1) as instructions executable by the at least one processor 102.

[0071] At the block 300, the at least one non-transitory computer-readable medium 116 stores the instructions that, when executed, configure the at least one processor 102 to receive an input content representing a visual scene. Specifically, the input content may be received by the at least one computing device 101 using the one or more input devices 106 or from the database 118 over the network 114.

[0072] If the input content is a still image, then at block 310, weights associated with dynamic effects on VAM may be set to zero. This is because increase preferences due to relative temporal contrast change (e.g., flashing) or relative motion, etc. may be irrelevant for the still image. This may ensure the VAM focuses only on relevant static features of the still image.

[0073] If the input content is a video, then at block 320, weights associated with dynamic effects on the VAM may be set appropriately, such as increase preferences due to relative temporal contrast change (e.g., flashing) or relative motion, etc. The dynamic effects like sudden movements or flashes may be powerful attention-grabbing features. Accordingly, the VAM may assign them a higher saliency score or weight during analysis, which may ensure the VAM appropriately considers these temporal features.

[0074] At block 400, input of user behavior and / or goal data may be received. In some embodiments, such input may include explicit goal data input, such as asking the user if they have a current goal and if so, what it is. In some embodiments, such input may include inferred goal and / or behavior data, such as data pertaining to their current actions.

[0075] At block 500, a user goal may be identified (if applicable) based on the input of user behavior and / or goal data. For example, if the calendar events indicate an upcoming meeting about a specific project, the process 250 may infer that the user's visual search on a related document is goal-oriented.

[0076] At block 600, image analytics may utilize processing of using algorithms and software to interpret visual information in the still images or the videos. This may include image segmentation, object detection, feature extraction, object tracking, etc.

[0077] At block 700, scene understanding and assessment of expectations for scene may be utilized. This may help to obtain an available information related to the visual scene. Scene understanding may be more complex as it may require additional processing to evaluate relationships between objects (or features) in the visual scene. This may often require use of more advanced deep learning and computer vision techniques to make sense of the visual scene and to identify outliers or anomalies in the visual scene. In terms of visual perception and visual attention, much of what matters depends on relative contrast. This is true in terms of visual contrast in edges, in terms of temporal changes, but also in terms of anomaly detection and impact on visual attention. This may also involve evaluating expectations for the visual scene.

[0078] At block 800, the VAM is determined based on the available information. The VAM weights may then be adjusted based on the blocks 300, 400, 500, and 600. As noted above, the block 300 impacts weights associated with temporal and dynamic features from the blocks 600 and 700 (image analytics and scene understanding) regardless of presence of any user goal or mindset still would impact VAM weights (such as increase weight of novel features and object). The outputs of the blocks 600 and 700 would further interact with the block 500 (user goals) in that features consistent with the user goal would receive increased VAM weights.

[0079] At block 900, VAM analysis may be performed based on current input and weights to determine saliency predictions. The VAM analysis may be performed to generate the predicted visual saliency for the regions of the visual scene using the determined VAM.

[0080] At block 1000, an output of the VAM analysis may then be fed into a larger system to compare the predicted saliency with an advertiser’ s / designer’s goals. In some embodiments, the larger system may generate or recommend scene modifications to achieve the advertiser’ s / designer’s goals. In some embodiments, the scene modifications may be constrained to selected color or text change modes, including hue, brightness, saturation, font type, font size, and text orientation, according to user defined value ranges or discrete allowed values. This may allow the user to ensure that automated scene modifications remain within brand guidelines or aesthetic preferences.

[0081] This adaptability may enhance prediction accuracy by ensuring that the most appropriate model is deployed for a given context. For example, determining the goal-oriented top down VAM for an active search task, which an existing conventional bottom up VAM would fail to predict correctly. Furthermore, this dynamic determination may lead to greater computational efficiency, as the at least one non-transitory computer-readable medium 116 can intelligently allocate resources by choosing a less-intensive model when complex goalinference is unnecessary.

[0082] The system 100, the at least one non-transitory computer-readable medium 116, and the method 200 may intelligently switch between the VAMs, such as bottom up, top down, and hybrid VAMs to predict visual attention and evaluate the saliency for the regions of the visual scene, such as the still image, the video, design, etc. The predicted visual attention and the evaluated saliency for the regions of the visual scene may be more accurate across a wide range of visual scenes, media types, and user states. This is because the system 100, the at least one non-transitory computer-readable medium 116, and the method 200 may determine when to use a particular VAM for a given application. Thus, the at least one non-transitory computer-readable medium 116, and the method 200 may ensure an appropriate VAM is deployed for a given context.

[0083] For example, if based on a direct input or any indirect data, the system 100, the at least one non-transitory computer-readable medium 116, and the method 200 may use the top down VAM or hybrid VAM. If unsure, the system 100, the at least one non-transitory computer-readable medium 116, and the method 200 may ask the user if they have a goal or a task in mind and if so, what it is. Otherwise, the system 100, the at least one non-transitory computer-readable medium 116, and the method 200 may default to using the bottom up VAM. That is, if the system 100, the at least one non-transitory computer-readable medium 116, and the method 200 have enough information to suggest that the user has a specific goal or a task in mind, they may use the top down VAM and weigh areas and objects associated with that task more heavily in terms of visual saliency. If the system 100, the at least one non-transitory computer-readable medium 116, and the method 200 cannot infer a specific user goal or a task, then it may assume passive viewing and use the bottom up VAM. Furthermore, this intelligent switching or dynamic determination may allow the at least one non-transitory computer-readable medium 116 to allocate computational resources by choosing a less-intensive VAM when a complex goal-inference is unnecessary, thereby improving overall efficiency.

[0084] FIG. 5 illustrates an exemplary embodiment of a segmented entity that is constrained to specific hue values, according to an embodiment of the present disclosure. FIG.5 shows a first version 1202 of a logo having a first hue value and a second version 1204 of the logo having a second, different hue value. For example, the first version 1202 may have a red hue, while the second version 1204 may have a black hue.

[0085] Embodiments of the present disclosure may allow users to define constraints whereby a color change is allowed, but the color change may only take specific values provided by the user. For example, a segment that represents a company logo, such as the logo shown in FIG. 5, may be changed from the first version 1202 to the second version 1204 only, where each version corresponds to a specific hue value that is permitted by the company. Thesespecific values may be provided on demand by the user when the constraint is created, or the values may be provided by the user previously and saved to a user profile for easier access and persistence.

[0086] This functionality may be applicable in scenarios where advertisements must adhere to region-specific marketing rules. For example, physical products made by one company and sold in a first country and a second country may be distinguished by a specific set of colors for the first country and a completely different set of colors for the second country. By using specific value constraints, advertisements for these products in each respective country may be configured to only show the product colors available for sale in that specific country.

[0087] FIGS. 6A and 6B illustrate exemplary embodiments where a physical product is distinguished by specific color values based on production years, according to embodiments of the present disclosure. Specifically, FIG. 6A shows a first version 1302 of a bicycle corresponding to a first production year and FIG. 6B shows a second version 1304 of the same bicycle model corresponding to a second, different production year.

[0088] Referring to FIGS. 6 A and 6B, the specific value constraints are applicable to scenarios where a physical product, made and sold in one country with the same name across multiple years, is distinguished by a new set of specific colors for each new production year. For example, the first version 1302 of the bicycle may be a model produced in 2023, while the second version 1304 of the bicycle may be the same model produced in 2021. In this embodiment, the color offered for the 2021 model is not available for the 2023 model, and vice versa. Similar types of specific value constraints may also be placed on other color change types, such as saturation and brightness, to allow users to precisely control how their content is modified.

[0089] FIG. 7 illustrates an exemplary user interface 1500 displaying application of different modification constraints and visual attention goals to multiple areas of interest, according to an embodiment of the present disclosure. Specifically, FIG. 7 shows the user interface 1500 displaying an input image 1502, a first area of interest (AOI) 1504, a second AOI 1506, a third AOI 1508, and a segment 1510 within the first AOI 1504.

[0090] In embodiments where two or more AOIs are defined or two or more segments are created, the system 100 allows the user to select two or more of the AOIs or segments and apply the same modification constraints or define a single visual attention goal that encompasses all selected regions. As shown in the exemplary scene, the user defined the first AOI 1504, the second AOI 1506, and the third AOI 1508. The system 100 also may have identified one or more of these as distractor regions. The user may select the first AOI 1504,which contains the segment 1510, and specify that a downstream optimization process should find ways to increase an attention score of the first AOI 1504 by modifying the hue, saturation, and brightness of the enclosed content. In some examples, the segment 1510 may be highlighted in a distinguishing color, e.g., green. Similarly, the AOIs may be highlighted in a distinguishing color.

[0091] Subsequently, the user may select the second AOI 1506 and specify that no changes should be made. Finally, the user may select the third AOI 1508 and specify that the attention score for the third AOI 1508 should be reduced by modifying the hue, saturation, and brightness of content within the third AOI 1508. These goals and constraints matriculate automatically to the segments enclosed within each respective AOI. As shown, a single AOI, such as the first AOI 1504, may correspond to one or multiple segments. This capability enables users to work faster and more efficiently when providing inputs that are redundant across multiple regions.

[0092] FIG. 8 illustrates an exemplary user interface 1600 displaying a user-defined gaze sequence goal that spans multiple areas of interest (AOI), according to an embodiment of the present disclosure. Specifically, FIG. 8 shows the user interface 1600 displaying an input image 1602, a first area of interest (AOI) 1604, a second AOI 1606, a first gaze sequence input 1608, and a second gaze sequence input 1610.

[0093] The ability to select multiple AOIs or segments enables the user to define gaze sequence goals that encompass two or more regions. In the exemplary scene shown, the user defined the first AOI 1604 and the second AOI 1606. The user further defines a gaze sequence goal by providing the first gaze sequence input 1608, represented by a numbered circle, and the second gaze sequence input 1610, also represented by a numbered circle. In this nonlimiting example, the user may wish a first predicted gaze sequence point to fall somewhere inside the first AOI 1604, and a second predicted gaze sequence point to fall somewhere inside the second AOI 1606. In some embodiments, the user may instead require that one or both gaze sequence points fall inside the specific segment within each respective AOI. Through the user interface 1600, the user may specify this goal, for example, by toggling a button to indicate that a gaze sequence goal is being created, clicking on each AOI or segment in the order of the desired gaze sequence, and then un-toggling the gaze sequence mode button. The at least one processor 102 may populate a gaze sequence goal using these inputs for a downstream optimization process.

[0094] FIG. 9 illustrates an exemplary user interface 1700 displaying a user-defined gaze sequence goal that spans multiple areas of interest without a specified order, according to an embodiment of the present disclosure. Specifically, FIG. 9 shows the user interface 1700displaying an input image 1702, a first area of interest (AOI) 1704, a second AOI 1706, a third AOI 1708, and a fourth AOI 1710.

[0095] The ability to select multiple AOIs or segments may also simplify defining a gaze sequence goal that does not include a specific gaze sequence order. As shown in the exemplary scene, the user selected the first AOI 1704, the second AOI 1706, the third AOI 1708, and the fourth AOI 1710. If the user wishes the first through the fourth predicted gaze sequence points to fall somewhere within these four AOIs but does not require the gaze points in a specific order, the user may select all four AOIs and specify this collective goal. The selection may be made, for example, by clicking each AOI individually while holding a specific key, or by using a drag-and-draw function to create a box that encloses all four AOIs. Notably, without specifying an order, a gaze sequence goal such as this example may be satisfied without requiring one gaze sequence point to fall in each region.

[0096] In some embodiments, if one or more gaze sequence goals are defined but cannot be achieved by modified scenes produced by scene optimization processes, a best modified scene or a set of best modified scenes as alternative solutions may be recommended. Although none of these alternative modified scenes meet all the gaze sequence goals, providing the alternative modified scenes to the user may help the user to identify what content in the scene should change or what is preventing the goals from being met. Algorithms such as simulated annealing optimization, greedy optimization, and weighted scoring systems may be used to determine which modified scenes come closest to achieving the gaze sequence goals.

[0097] FIGS. 10 A- 10C illustrate exemplary user interfaces 2800, 2810, 2820 describing generation of an attention-weighted image from an original scene, according to an embodiment of the present disclosure. Specifically, FIG. 10A shows an original image 2802 provided to the system 100 as input. FIG. 10B shows a predicted visual attention heatmap 2804, generated by the VAM 204, overlaid on the original image 2802. FIG. 10C shows a convolved image 2806, which is the result of combining the original image 2802 with the attention heatmap 2804.

[0098] Referring to FIGS. 28A-28C, the identification of distractor regions may be achieved through methods that build upon predictions made by the visual attention and object segmentation models. The VAM 204 is configured to predict where the attention will go across an entire scene and produce the attention heatmap 2804, as shown in FIG. 10B. To create the convolved image 2806 shown in FIG. 10C, the values from the attention heatmap 2804 are transformed to a [0, 1] scale and then multiplied with individual pixel values of the original image 2802. This process produces a convolved image that effectively weighs a content of the original image by the predicted attention.

[0099] In some embodiments, the transformation of the heatmap values to the [0, 1] scale may be performed in a linear manner, for example, by dividing each value in the heatmap 2804 by a maximum value found within the heatmap 2804. In some embodiments, the transformation may be performed in a non-linear manner, for example, by passing a pixel-level attention values to a sigmoid function. An inflection point of the sigmoid function may be set based on the visual attention goals of the user, allowing for a more customized weighting of the attention values.

[0100] Unless otherwise indicated, all numbers expressing feature sizes, amounts, and physical properties used in the specification and claims are to be understood as being modified by the term “about.” Accordingly, unless indicated to the contrary, the numerical parameters set forth in the foregoing specification and attached claims are approximations that can vary depending upon the desired properties sought to be obtained by those skilled in the art utilizing the teachings disclosed herein.

[0101] Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and / or equivalent implementations can be substituted for the specific embodiments shown and described without departing from the scope of the present disclosure. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that this disclosure be limited only by the claims and the equivalents thereof.

Claims

CLAIMSWhat is claimed is:

1. At least one non-transitory computer-readable medium storing instructions that, when executed, configure at least one processor toreceive an input content representing a visual scene;process the input content to obtain available information related to the visual scene; determine a visual attention model (VAM) based on the available information; and generate predicted visual saliency for regions of the visual scene using the determined VAM.

2. The at least one non-transitory computer-readable medium of claim 1 wherein processing the input content includes performing one or more image analytics operations including object detection, segmentation, feature extraction, or object tracking.

3. The at least one non-transitory computer-readable medium of claim 1 wherein the available information includes explicit user inputs including voice input, text input, graphical interface input, or search terms.

4. The at least one non-transitory computer-readable medium of claim 1 wherein the available information includes behavioral indicators including eye movement patterns, gaze direction, real world interactions, or movement patterns.

5. The at least one non-transitory computer-readable medium of claim 1 wherein determining the visual attention model comprises selecting between a bottom up model, a top down model, or a hybrid model based on whether the user is determined to have a specific task or goal.

6. The at least one non-transitory computer-readable medium of claim 5 wherein determining whether the user has a specific task or goal includes analyzing contextual data including to do lists, calendar events, or user profiles.

7. The at least one non-transitory computer-readable medium of claim 1 wherein determining the visual attention model includes applying collaborative filtering based on behavioral patterns of users having similar profiles or actions.

8. The at least one non-transitory computer-readable medium of claim 1 includes further instructions for adjusting parameters of the determined VAM based on semantic relationships among scene components or based on matches between scene components and features associated with an inferred user goal.

9. The at least one non-transitory computer-readable medium of claim 1 wherein generating the predicted visual saliency includes weighting whole objects based on similarity between object features and features associated with an inferred user goal.

10. The at least one non-transitory computer-readable medium of claim 1 includes further instructions for training one or more of the visual attention models using visual attention data obtained from eye-tracking devices in combination with task-related contextual information.

11. At least one system includesat least one computing device that includes one or more processors; andat least one memory coupled to at least one of the one or more processors, wherein the at least one memory includes instructions that configure the at least one computing device to receive an input content representing a visual scene;process the input content to obtain available information related to the visual scene; determine a visual attention model (VAM) based on the available information; and generate predicted visual saliency for regions of the visual scene using the determined VAM.

12. The at least one non-transitory computer-readable medium of claim 1 wherein processing the input content includes performing one or more image analytics operations including object detection, segmentation, feature extraction, or object tracking.

13. The at least one non-transitory computer-readable medium of claim 1 wherein the available information includes explicit user inputs including voice input, text input, graphical interface input, or search terms.

14. The at least one non-transitory computer-readable medium of claim 1 wherein the available information includes behavioral indicators including eye movement patterns, gaze direction, real world interactions, or movement patterns.

15. The at least one non-transitory computer-readable medium of claim 1 wherein determining the visual attention model comprises selecting between a bottom up model, a top down model, or a hybrid model based on whether the user is determined to have a specific task or goal.

16. The at least one non-transitory computer-readable medium of claim 5 wherein determining whether the user has a specific task or goal includes analyzing contextual data including to do lists, calendar events, or user profiles.

17. A computer-implemented method includesreceiving an input content representing a visual scene;processing the input content to obtain available information related to the visual scene; determining a visual attention model (VAM) based on the available information; and generating predicted visual saliency for regions of the visual scene using the determined VAM.

18. The computer-implemented method of claim 17 wherein processing the input content includes performing one or more image analytics operations including object detection, segmentation, feature extraction, or object tracking.

19. The computer-implemented method of claim 17 wherein the available information includes explicit user inputs including voice input, text input, graphical interface input, or search terms.

20. The computer-implemented method of claim 17 wherein the available information includes behavioral indicators including eye movement patterns, gaze direction, real world interactions, or movement patterns.

Images