Systems, media, and methods for identification, characterization, and modification of visual attention areas of interest, distractors, non-salient regions, and linked regions

Abstract

A computer‑implemented method includes receiving image data representing a scene and predicting regions associated with visual attention using a visual attention model (VAM) and identifying one or more regions that include at least one of (i) one or more areas of interest (AOI), (ii) one or more distractor regions, and (iii) one or more non‑salient regions (NSR). The method further includes generating segmented entities corresponding to the regions by utilizing one or more models that include a segmentation model, a classification model, or both, on the image data and determining visual or structural relationships between a combination of regions and segmented entities and scene contents excluding segmented entities. The method also includes generating or recommending scene modifications to one or more of the identified regions to achieve a user‑defined visual attention goal.

Description

SYSTEMS, MEDIA, AND METHODS FOR IDENTIFICATION, CHARACTERIZATION, AND MODIFICATION OF VISUAL ATTENTION AREAS OF INTEREST, DISTRACTORS, NONSALIENT REGIONS, AND LINKED REGIONSTechnical Field

[0001] The present disclosure generally relates to systems, media, and methods for identification, characterization, and modification of visual attention areas of interest, distractors, non-salient regions, and linked regions.Background

[0002] Prediction of visual attention in a visual scene, such as determining which parts of the visual scene are most salient, looked at, or attended to, and a likely order in which those elements will be viewed, is a critical component in various fields of analysis and design. For example, such analysis is applicable to merchandise placement, environmental design, safety assessments, web page layout, advertising effectiveness, and product marketing. Designers may want to identify components in an advertisement or design that they would like to get more attention (e.g., brand logos, product image, and key message items).

[0003] In many visual attention analysis tools, a user may specify one or more areas of interest (AOIs), which correspond to regions in a scene for which a higher degree of visual attention is desired. These AOIs may include, for instance, brand logos, product images, or key textual messages. Conventional visual attention analysis tools are capable of analyzing a visual scene to predict which regions will exhibit high saliency and likely to capture more attention, and can report visual features within a specified AOI that contribute to the predicted saliency score for that AOI.

[0004] However, the changes or modifications that are required to be made to the visual scene to meet a visual hierarchy and attention goals are not estimated by any existing visual attention analysis tool, and in practice may only be estimated through a trial and error edit / evaluation process. In this inefficient process, the user makes an edit to the visual scene, re-evaluates the visual scene with the visual attention analysis tool, and repeats the process until the desired attention goals are met. In general, it may be possible to meet the attention goals associated with an AOI by modifying a content within the AOI or by modifying the content outside the AOI.

[0005] Furthermore, existing tools do not provide or often fail to provide a causal explanation for why certain regions outside of the designated AOIs (herein referred to as “distractors”) capture a high degree of visual attention, sometimes at the expense of the AOIs. A user may also find it difficult to ascertain from the existing tools which specific visual attention capture mechanisms, such as red / green color contrast, blue / yellow color contrast, luminance contrast, or edge density are being under-utilized or over-utilized within the visual scene.Summary

[0006] In one embodiment, at least one non-transitory computer-readable medium storing instructions that, when executed, configure at least one processor for receiving image data representing a scene and predicting regions associated with visual attention using a visual attention model (VAM) and identifying one or more regions that include at least one of (i) one or more areas of interest (AOI), (ii) one or more distractor regions, and (iii) one or more non-salient regions (NSR). The at least one non-transitory computer-readable medium includes further instruction for generating segmented entities corresponding to the regions by utilizing one or more models that include a segmentation model, a classification model, or both, on the image data and determining visual or structural relationships between a combination of regions and segmented entities and scene contents excluding segmented entities. The at least one non-transitory computer-readable medium also includes further instructions for generating or recommending scene modifications to one or more of the identified regions to achieve a user-defined visual attention goal.

[0007] In another embodiment, at least one system includes at least one computing device that includes one or more processors and at least one memory coupled to at least one of the one or more processors, wherein the at least one memory includes instructions that configure the at least one computing device to receive image data representing a scene and predict regions associated with visual attention using a visual attention model (VAM) and identifying one or more regions that include at least one of (i) one or more areas of interest (AOI), (ii) one or more distractor regions, and (iii) one or more non-salient regions (NSR). The at least one system includes further instructions to generate segmented entities corresponding to the regions by utilizing one or more models that include a segmentation model, a classification model, or both, on the image data as well as determine visual or structural relationships between a combination of regions and segmented entities and scene contents excluding segmented entities. The at least one system also includes further instructions to generate or recommend scene modifications to one or more of the identified regions to achieve a user-defined visual attention goal.

[0008] In yet another embodiment, a computer-implemented method includes receiving image data representing a scene and predicting regions associated with visual attention using a visual attention model (VAM) and identifying one or more regions that include at least one of (i) one or more areas of interest (AOI), (ii) one or more distractor regions, and (iii) one or more non-salient regions (NSR). The method further includes generating segmented entities corresponding to the regions by utilizing one or more models that include a segmentation model, a classification model, or both, on the image data and determining visual or structural relationships between a combination of regions and segmented entities and scene contents excluding segmented entities. The method also includes generating or recommending scene modifications to one or more of the identified regions to achieve a user-defined visual attention goal.

[0009] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims.Brief Description of Drawings

[0010] Exemplary embodiments disclosed herein may be more completely understood in consideration of the following detailed description in connection with the following figures. The figures are not necessarily drawn to scale. Like numbers used in the figures refer to like components. However, it will be understood that the use of a number to refer to a component in a given figure is not intended to limit the component in another figure labeled with the same number.

[0011] FIG. 1 illustrates a schematic block diagram of at least one system for enhancing a scene, according to an embodiment of the present disclosure;

[0012] FIG. 2 illustrates an exemplary block diagram of at least one processor of at least one computing device of the at least one system, according to an embodiment of the present disclosure;

[0013] FIG. 3 illustrates a flowchart of a method for enhancing a scene, according to an embodiment of the present disclosure;

[0014] FIG. 4 illustrates a flowchart of an exemplary process for automatically triggering execution of one or more segmentation models, according to an embodiment of the present disclosure;

[0015] FIGS. 5 A and 5B illustrate exemplary user interfaces for displaying automatic execution of the one or more segmentation models, according to an embodiment of the present disclosure;

[0016] FIGS. 6A-6C illustrate an exemplary comparison of different outputs of the one or more segmentation models, according to an embodiment of the present disclosure;

[0017] FIGS. 7A and 7B illustrate exemplary user interfaces displaying automatic classification of segmented entities, according to an embodiment of the present disclosure;

[0018] FIGS. 8 A and 8B illustrate exemplary user interfaces displaying manual reclassification of a segmented entity, according to an embodiment of the present disclosure;

[0019] FIG. 9 illustrates a flowchart of an exemplary process for refining a segmented entity, according to an embodiment of the present disclosure;

[0020] FIGS. 10A and 10B illustrate exemplary user interfaces where user inputs create a new segmented entity linked with an original segment, according to embodiments of the present disclosure;

[0021] FIGS. 11A and 11B illustrate exemplary user interfaces where the user inputs expand a scope of an existing segment, according to embodiments of the present disclosure;

[0022] FIG. 12 illustrates an exemplary embodiment of a segmented entity that is constrained to specific hue values, according to an embodiment of the present disclosure;

[0023] FIGS. 13A and 13B illustrate exemplary embodiments where a physical product is distinguished by specific color values based on production years, according to the present disclosure;

[0024] FIG. 14 illustrates exemplary feedback systems for obtaining user preference feedback, according to an embodiment of the present disclosure;

[0025] FIG. 15 illustrates an exemplary user interface displaying application of different modification constraints and visual attention goals to multiple areas of interest, according to an embodiment of the present disclosure;

[0026] FIG. 16 illustrates an exemplary user interface displaying a user-defined gaze sequence goal that spans multiple areas of interest, according to an embodiment of the present disclosure;

[0027] FIG. 17 illustrates an exemplary user interface displaying a user-defined gaze sequence goal that spans multiple areas of interest without a specified order, according to an embodiment of the present disclosure;

[0028] FIG. 18 illustrates an exemplary user interface displaying output of a scene optimization process, according to an embodiment of the present disclosure;

[0029] FIGS. 19A and 19B illustrate exemplary user interfaces showing a recommended scene modification from an optimization process, according to an embodiment of the present disclosure;

[0030] FIGS. 20A and 20B illustrate exemplary user interfaces displaying a visual representation of a result from a sensitivity analysis, according to an embodiment of the present disclosure;

[0031] FIGS. 21A and 21B illustrate exemplary user interfaces displaying a new recommended scene modification resulting from a re-executed optimization process, according to an embodiment of the present disclosure;

[0032] FIGS. 22A and 22B illustrate exemplary user interfaces displaying visual representation of a result from a sensitivity analysis, according to another embodiment of the present disclosure;

[0033] FIGS. 23A-23C illustrate exemplary user interfaces displaying a process of segmenting and refining entities within a scene, according to an embodiment of the present disclosure;

[0034] FIGS. 24A and 24B illustrate exemplary user interfaces for displaying different layers of a visual attention software (VAS) analysis, according to an embodiment of the present disclosure;

[0035] FIGS. 25A-25C illustrate exemplary user interfaces for specifying attention goals and modification constraints for segmented entities, according to an embodiment of the present disclosure;

[0036] FIGS. 26A and 26B illustrate exemplary user interfaces for displaying results of an optimization process, according to an embodiment of the present disclosure.

[0037] FIG. 27 illustrates a flowchart of an exemplary process for optimizing a scene based on user-defined goals, according to an embodiment of the present disclosure;

[0038] FIGS. 28A-28C illustrate exemplary user interfaces describing generation of an attention-weighted image from an original scene, according to an embodiment of the present disclosure;

[0039] FIGS. 29A and 29B illustrate exemplary user interfaces describing segmentation of visually salient objects identified from a convolved image, according to an embodiment of the present disclosure;

[0040] FIGS. 30A and 3 OB illustrate exemplary user interfaces describing an alternative process for segmenting visually salient objects using a saliency threshold, according to an embodiment of the present disclosure;

[0041] FIGS. 31A and 3 IB illustrate exemplary user interfaces describing identification of distractor regions using predicted attention hotspots, according to an embodiment of the present disclosure;

[0042] FIGS. 32A-32C illustrate a series of exemplary images and associated visual attention analysis results, demonstrating effects of modifying different regions within a scene, according to an embodiment of the present disclosure;

[0043] FIGS. 33A and 33B illustrate a series of exemplary images and associated visual attention analysis results, demonstrating effects of modifying non-salient regions and all regions combined, according to an embodiment of the present disclosure; and

[0044] FIG. 34 illustrates an exemplary user interface describing application of the scale -invariant feature transform (SIFT) algorithm for detecting and locating similar image regions, according to an embodiment of the present disclosure.Detailed Description

[0045] In the following description, reference is made to the accompanying figures that form a part thereof and in which various embodiments are shown by way of illustration. It is to be understood that other embodiments are contemplated and may be made without departing from the scope or spirit of the present disclosure. The following detailed description, therefore, is not to be taken in a limiting sense.

[0046] In the following disclosure, the following definitions are adopted.

[0047] As used herein, all numbers should be considered modified by the term “about”. As used herein, “a,” “an,” “the,” “at least one,” and “one or more” are used interchangeably.

[0048] As used herein as a modifier to a property or attribute, the term “generally”, unless otherwise specifically defined, means that the property or attribute would be readily recognizable by a person of ordinary skill but without requiring absolute precision or a perfect match (e.g., within + / - 20 % for quantifiable properties).

[0049] The term “substantially”, unless otherwise specifically defined, means to a high degree of approximation (e.g., within + / - 10% for quantifiable properties) but again without requiring absolute precision or a perfect match.

[0050] The term “about”, unless otherwise specifically defined, means to a high degree of approximation (e.g., within + / - 5% for quantifiable properties) but again without requiring absolute precision or a perfect match.

[0051] As used herein, the terms “first” and “second” are used as identifiers. Therefore, such terms should not be construed as limiting of this disclosure. The terms “first” and “second” when usedin conjunction with a feature or an element can be interchanged throughout the embodiments of this disclosure.

[0052] As used herein, “at least one of A and B” should be understood to mean “only A, only B, or both A and B”.

[0053] As used in some embodiments herein, the term “user” generally refers to a person who wishes to enhance or modify a scene or an image. The user may include an owner of a product or a service provider.

[0054] As used in some embodiments herein, the term “entity” generally refers to a thing or an area or a region within a scene, or regions within a scene that may be analyzed.

[0055] As used in some embodiments herein, the term “saliency metric” generally refers to an amount of attention attracted at a time of viewing a scene or an image.

[0056] As used in some embodiments herein, the term “attention score” generally refers to an amount of attention given to a specific area in an image or a scene.

[0057] As used in some embodiments herein, the term “gaze sequence” generally refers to an order or a sequence in which a viewer is likely to view different regions of a visual representation.

[0058] Prediction of visual attention in a visual scene, such as determining which parts of the visual scene are most salient, looked at, or attended to, and a likely order in which those elements will be viewed, is a critical component in various fields of analysis and design. For example, such analysis is applicable to merchandise placement, environmental design, safety assessments, web page layout, advertising effectiveness, and product marketing. Designers may want to identify components in an advertisement or design that they would like to get more attention (e.g., brand logos, product image, and key message items).

[0059] In many visual attention analysis tools, a user may specify one or more areas of interest (AOIs), which correspond to regions in a scene for which a higher degree of visual attention is desired. These AOIs may include, for instance, brand logos, product images, or key textual messages. Conventional visual attention analysis tools are capable of analyzing a visual scene to predict which regions will exhibit high saliency and likely to capture more attention, and can report visual features within a specified AOI that contribute to the predicted saliency score for that AOI.

[0060] However, the changes or modifications that are required to be made to the visual scene to meet a visual hierarchy and attention goals are not estimated by any existing visual attention analysis tool, and in practice may only be estimated through a trial and error edit / evaluation process. In this inefficient process, the user makes an edit to the visual scene, re-evaluates the visual scene with the visual attention analysis tool, and repeats the process until the desired attention goals are met. In general, it may be possible to meet the attention goals associated with an AOI by modifying a content within the AOI or by modifying the content outside the AOI.

[0061] Furthermore, existing tools do not provide or often fail to provide a causal explanation for why certain regions outside of the designated AOIs (herein referred to as “distractors”) capture a highdegree of visual atention, sometimes at the expense of the AOIs. A user may also find it difficult to ascertain from the existing tools which specific visual atention capture mechanisms, such as red / green color contrast, blue / yellow color contrast, luminance contrast, or edge density are being under-utilized or over-utilized within the visual scene. Such information would be valuable for strategically modifying the saliency of an AOI.

[0062] Additionally, a limitation of conventional approaches is a narrow focus on modifying only the user-specified AOIs. This approach fails to consider the visual scene holistically. Optimizing visual atention may require not only enhancing an AOI, but also strategically de -emphasizing identified distractor regions that compete for atention or modifying non-salient regions (NSRs) to alter the overall context.

[0063] According to aspects of the disclosure, at least one non-transitory computer-readable medium stores instructions that, when executed, configure at least one processor for receiving image data representing a scene; predicting regions associated with visual atention using a visual atention model (VAM) to identify one or more regions that include at least one of (i) one or more areas of interest (AOI), (ii) one or more distractor regions, and (iii) one or more non-salient regions (NSR); generating segmented entities corresponding to the one or more regions by utilizing one or more models that include a segmentation model, a classification model, or both, on the image data; determining visual or structural relationships between the segmented entities; and generating or recommending scene modifications to one or more of the identified regions to achieve a user-defined visual atention goal.

[0064] The at least one processor of the present disclosure may perform operations on the image data representing the scene, including the application of the visual atention model (VAM). Application of the VAM provides a quantitative prediction of human perceptual focus by identifying the one or more regions, such as the areas of interest (AOI), the distractor regions, and the non-salient regions (NSR), thereby establishing an objective baseline for analysis that is independent of subjective human assessment. The at least one processor is further configured to generate the segmented entities corresponding to the one or more regions through use of the segmentation and classification models, which enables a granular, object-level examination of the scene composition. The subsequent determination of the visual and structural relationships between the segmented entities may enable determination of a visual hierarchy of the scene. The generation of the specific scene modifications based on the visual and structural relationships provides actionable, data-driven recommendations for achieving a user-defined visual atention goal, enabling a systematic optimization of the scene to direct the gaze of a viewer.

[0065] FIG. 1 illustrates a schematic block diagram of at least one system 100 for enhancing a scene, according to an embodiment of the present disclosure. The at least one system 100 is configured to receive image data representing a scene based on user goals of a user. A user goal refers to a user-desired manner in which a viewer atends to entities within a scene. For example, the user goals mayrelate to specific objects that are attended to in addition to collection of objects that are deemed unimportant or even detrimental, specific objects attended to in a particular sequence or at a particular time, or specific objects attended to from a particular viewing point, and the like. The system 100 is configured to perform a holistic analysis of the scene, identifying not only user-defined areas of interest (AOIs), but also automatically identifying distractor regions that capture undue attention and nonsalient regions (NSRs) that form a background. The holistic analysis allows for comprehensive scene modification to achieve the user goals.

[0066] The at least one system 100 includes at least one computing device 101, through which embodiments of the present disclosure may be implemented, such as those depicted and / or described in other figures herein. The at least one computing device 101, as described herein, is one example of a suitable computing device and does not suggest any limitation on the scope of any embodiments presented. Nothing illustrated or described with respect to the computing device 101 should be interpreted as being required or as creating any type of dependency with respect to any element or plurality of elements.

[0067] In various embodiments, the computing device 101 may include, but need not be limited to, a desktop, laptop, server, client, tablet, smartphone, computing cloud, or any other type of device that can utilize data. In an embodiment, the at least one computing device 101 includes at least one processor 102 and at least one memory 110. The at least one memory 110 includes a non-volatile memory 108 (e.g., ROM, flash memory, etc.), a volatile memory (e.g., RAM, etc.) 108’, and / or a combination thereof. The term “at least one processor 102” is interchangeably referred to herein as “one or more processors 102”. Specifically, the at least one computing device 101 includes the one or more processors 102. The at least one memory 110 is coupled to at least one of the one or more processors 102. The at least one memory 110 includes instructions that configure the at least one computing device 101 for enhancing the scene.

[0068] The at least one computing device 101 may include one or more displays 104, a display hardware, and / or output devices such as, for example, AR / VR / MR / XR hardware, monitors, speakers, headphones, projectors, wearable-displays, holographic displays, and / or printers. The output devices may further include, for example, displays and / or speakers, devices that emit energy (radio, microwave, infrared, visible light, ultraviolet, x-ray and gamma ray), electronic output devices (Wi-Fi, radar, laser, etc.), audio (of any frequency), and the like.

[0069] The at least one computing device 101 further include one or more input devices 106 which may include, by way of example, any type of mouse, a keyboard, a disk / media drive, a memory stick / thumb-drive, a memory card, a pen, a touch-input device, a biometric scanner, a gaze and / a or blink tracker, a tracker, a voice / auditory input device, a motion-detector, a camera, a scale, and any device capable of measuring data, such as motion data (e.g., an accelerometer, a GPS, a magnetometer, a gyroscope, etc.), biometric data (e.g., blood pressure, pulse, heart rate, perspiration, a temperature, a voice, a facial-recognition, a motion / gesture tracking, a gaze tracking, an iris or other types of eyerecognition, a hand geometry, an oxygen saturation, a glucose level, a fingerprint, DNA, dental records, a weight, or any other suitable type of biometric data, etc.), video / still images, and an audio (including human-audible and human-inaudible ultrasonic sound waves). The one or more input devices 106 may include any type of device capable of receiving data, whether from another device, a visual and / or an audio data captured from the real world, an object detection data, and the like. The one or more input devices 106 may include cameras (with or without audio recording), such as digital and / or analog cameras, still cameras, video cameras, thermal imaging cameras, infrared cameras, cameras with a charge-couple display, night-vision cameras, three-dimensional cameras, webcams, audio recorders, and the like.

[0070] The at least one computing device 101 further includes a network interface 112 that can facilitate communications over a network 114 with other data sources, such as a database 118, via wires, a wide area network, a local area network, a personal area network, a cellular network, a satellite network, and the like. Suitable local area networks may include wired Ethernet and / or wireless technologies such as, for example, wireless fidelity (Wi-Fi). Suitable personal area networks may include wireless technologies such as, for example, infrared data association (IrDA), Bluetooth®, Wireless USB, Z-Wave, ZigBee, and / or other near field communication protocols. Suitable personal area networks may similarly include wired computer buses such as, for example, a universal serial bus (USB) and a FireWire. Suitable cellular networks may include, but are not limited to, technologies such as long-term evolution (UTE), worldwide interoperability for microwave access (WiMAX), universal mobile telecommunications system (UMTS), code-division multiple access (CDMA), and global system for mobile communication (GSM).

[0071] The network interface 112 can be communicatively coupled to any device capable of transmitting and / or receiving data via one or more network(s). Accordingly, the network interface 112 may include a communication transceiver for sending and / or receiving any wired or wireless communication. For example, the network interface 112 may include an antenna, a modem, a local area network (LAN) port, a Wi-Fi card, a WiMax card, a mobile communications hardware, a nearfield communication hardware, a satellite communication hardware, and / or any wired or wireless hardware for communicating with other networks and / or devices.

[0072] The at least one computing device 101 further includes at least one non-transitory computer-readable medium 116. The at least one non-transitory computer-readable medium 116 may reside, for example, within the one or more input devices 106, the non-volatile memory 108, the volatile memory 108’, or any combination thereof. The at least one non-transitory computer-readable medium 116 can include tangible media that is able to store instructions associated with, or used by, a device or a system. The at least one non-transitory computer-readable medium 116, also referred to herein as a non-transitory computer-readable medium, includes, by way of non-limiting examples: random access memory (RAM), read only memory (ROM), cache, fiber optics, erasable programmable read-only memory (EPROM)ZFlash memory, CD / DVD / BD-ROM, hard disk drives, solid-statestorage, optical or magnetic storage devices, diskettes, electrical connections having a wire, or any combination thereof. The at least one non-transitory computer-readable medium 116 may also include, for example, a system or a device that is of a magnetic, optical, semiconductor, or electronic type. The at least one non-transitory computer-readable medium 116 excludes carrier waves and / or propagated signals taking any number of forms such as optical, electromagnetic, or combinations thereof.

[0073] The network interface 112 facilitates communication of the at least one computing device 101 with one or more remote devices, which may include, for example, client and / or server devices. The network interface 112 may also be described as a communications module, as these terms may be used interchangeably. The database 118 is depicted as being accessible over the network and may reside within a server, a cloud, or any other configuration to support being able to remotely access data and store data in the database 118.

[0074] FIG. 2 illustrates an exemplary block diagram of the at least one processor 102 of the at least one computing device 101 (shown in FIG. 1), according to an embodiment of the present disclosure. Referring to FIGS. 1 and 2, the at least one processor 102 is configured to receive the image data representing the scene. In various embodiments, the at least one processor 102 receives the image data from the one or more input devices 106 and presents an output on the one or more displays 104.

[0075] The at least one processor 102 includes a visual attention model (VAM) 204 for analyzing the image data and the scene, determining an attention score or map of the scene, and generating a VAM output. The VAM 204 analyzes the scene to identify one or more regions including user-defined AOIs, distractor regions, and non-salient regions (NSRs). In some embodiments, as depicted by the dashed lines, the at least one processor 102 includes one or more segmentation models 202 for identifying pixels associated with entities in an image or a video frame, for example, within a user-defined AOI or other identified regions. Specifically, the one or more segmentation models 202 may analyze the one or more regions and identify objects or entities therein.

[0076] The at least one processor 102 further includes one or more optimization models 206 for modifying the segmented entities within the one or more regions to achieve user goals, which may include optimizing the VAM output. Specifically, the one or more optimization models 206 are configured to modify the segmented entities within any of the identified regions (AOIs, distractors, or NSRs) to achieve the user-defined visual attention goals. In some embodiments, the at least one processor 102 further includes a classification model 208 for classifying each segmented entity to assign a class type (e.g., a text, an object, etc.). In some embodiments, the VAM 204, the one or more optimization model 206, the one or more segmentation models 202, and the classification model 208 may function iteratively to enhance the scene according to the user goals.

[0077] Note that the VAM 204, the one or more optimization models 206, the one or more segmentation models 202, and the classification model 208 may be implemented using one or more artificial intelligence (Al) or machine learning techniques. For such a purpose, the VAM 204, the oneor more optimization models 206, the one or more segmentation models 202, and the classification model 208 may be trained on one or more datasets stored locally in memory 110 or remotely in database 118. In some embodiments, the one or more segmentation models 202 may be trained over a dataset of various scenes or image data to identify and delineate a plurality of entities. The VAM 204 may be trained over a dataset to calculate attention metrics from scenes or image data, and in certain embodiments, may be trained over already segmented entities received from the one or more segmentation models 202. The VAM 204 may also be trained to specifically identify and differentiate between the one or more regions, i.e., AOIs, distractor regions, and NSRs. The one or more optimization models 206 may be trained to generate scene modifications based on outputs from the VAM 204 or other user-defined constraints. The operations of the VAM 204, the one or more optimization models 206, and the one or more segmentation models 202, and the classification model 208 will be discussed in further detail in the descriptions of subsequent figures.

[0078] FIG. 3 illustrates a flowchart of a method 300 for enhancing a scene, according to an embodiment of the present disclosure. Note that the method 300 is configured to be performed using the at least one system 100 shown in FIG. 1. Specifically, the method 300 is configuredto be performed by the at least one computing device 101 of the at least one system 100. More specifically, the at least one non-transitory computer-readable medium 116 is configured to store instructions that, when executed, configure the at least one processor 102 to perform steps of the method 300.

[0079] At operation 302, the method 300 includes receiving the image data representing the scene. The image data may be received by the at least one computing device 101 using the one or more input devices 106 or from the database 118 over the network 114.

[0080] At operation 304, the method 300 further includes predicting regions associated with visual attention using a visual attention model (VAM) (e.g., the VAM 204 shown in FIG. 2) to identify one or more regions that include at least one of (i) one or more user-defined areas of interest (AOI), (ii) one or more distractor regions, and (iii) one or more non-salient regions (NSR). The VAM may analyze the entire scene to produce an attention heatmap or saliency map. Based on this map, user inputs defining AOIs, and predefined criteria, the system 100 partitions the scene into the one or more regions for a holistic analysis.

[0081] In some embodiments, identification of the distractor regions further includes calculating overlaps between predicted gaze sequence points or attention hotspots and segmented objects, and determining a region as a distractor when overlap area thresholds or distance criteria are satisfied.

[0082] At operation 306, the method 300 further includes generating segmented entities corresponding to the identified regions by utilizing one or more models that include a segmentation model (e.g., the one or more segmentation models 202 shown in FIG. 2), a classification model (e.g., the classification model 208 shown in FIG. 2), or both, on the image data. For each identified region (AOIs, distractor regions, and NSRs), the system 100 executes the segmentation model to create aprecise pixel-level mask of objects or elements within that region, thereby generating segmented entities.

[0083] In some embodiments, the method 300 further includes determining, for each segmented entity, one or more visual attention metrics based on saliency predictions or gaze sequence points produced by the VAM 204. This provides a quantifiable baseline of visual hierarchy before any modifications are applied.

[0084] In some embodiments, execution of the segmentation model is triggered based on a timebased or an event-based condition, including initiating the segmentation automatically after a predetermined time passes from a last user modification of an AOI, or upon detection of a user event in a graphical interface. The automatic execution is designed to streamline a user workflow by reducing manual inputs.

[0085] In some embodiments, classifying segmented entities further includes applying multiple segmentation paradigms selected from instance segmentation, semantic segmentation, and panoptic segmentation, and merging their outputs to form a unified segmentation result (shown in FIGS. 6A-6C). This may allow the system 100 to leverage strengths of different models to achieve a highly accurate and comprehensive understanding of the scene.

[0086] In some embodiments, the method 300 further includes providing a user interface that displays the segmented entities and allows the user to refine segmentation boundaries by supplying inclusion point prompts, exclusion point prompts, or mask inputs from external tools, and automatically rerunning segmentation when new inputs are received. This feedback loop ensures high-fidelity segmentation aligned with user intent.

[0087] At operation 308, the method 300 further includes determining visual or structural relationships between the segmented entities. This step involves analyzing the segmented entities to identify relationships, such as identifying multiple instances of the same logo or text element that should be treated as linked. In some embodiments, determining visual or structural relationships between the segmented entities further includes identifying linked regions exhibiting visual or structural similarity. This allows the system 100 to recognize and group multiple instances of the same object, such as all product logos in a scene, ensuring they may be edited consistently as a single unit, which streamlines the user workflow.

[0088] At operation 310, the method 300 further includes generating or recommending scene modifications to one or more of the identified regions to achieve a user-defined visual attention goal. Based on the goals of the user (e.g., increase visual attention to an AOI, decrease attention to a distractor), an optimization model (e.g., the one or more optimization models 206) generates or recommends changes to visual properties of the segmented entities in any of the identified regions.

[0089] In some embodiments, generating the scene modifications further includes constraining modifications to selected color or text change modes, including hue, brightness, saturation, font type, font size, and text orientation, according to user defined value ranges or discrete allowed values. Thisallows the user to ensure that automated modifications remain within brand guidelines or aesthetic preferences.

[0090] Thus, the method 300 of the present disclosure provides a comprehensive process for holistic scene enhancement. By identifying and categorizing the one or more regions into areas of interests (AOIs), distractor regions, and non-salient regions (NSRs), the method 300 enables atargeted and highly effective approach to guiding visual attention. The use of automatic, trigger-based execution for segmentation maintains an efficient and streamlined user workflow, while the ability of the user to refine segments with intuitive inputs ensures high-fidelity masks crucial for precise modifications. This detailed scene decomposition allows the system 100 to not only enhance desired elements within the AOIs, but also to strategically diminish a visual impact of identified distractor regions, resulting in a more effective and compliant final design. Ultimately, the accurately segmented and classified entities serve as the foundation for advanced functionalities, providing the user with robust, data-driven tools for optimizing visual content.

[0091] In some embodiments, the method 300 further includes performing an analysis to identify linked regions of the scene exhibiting visual or structural similarity to a segmented entity, using one or more feature detection methods or neural network-based matching. This provides a solution for linked element editing by determining if an object to be edited has similar elements in the scene within a tolerance threshold and ensuring that similar or equivalent edits are applied to these linked elements. In some embodiments, determining the linked regions includes applying a similarity detection algorithm selected from image matching or feature detection techniques, such as scale invariant feature transform (SIFT), speeded up robust features (SURF), or deep learning approaches using convolutional neural network (CNN) feature matching to detect visually or structurally similar regions for synchronized editing.

[0092] In some embodiments, the method 300 further includes performing a sensitivity analysis to determine an effect of incremental modifications in hue, brightness, or saturation on at least one attention score derived from the VAM 204, and using the sensitivity information to refine subsequent modification recommendations. This provides the user with valuable insight into robustness of optimization and criticality of each visual parameter.

[0093] In some embodiments, the method 300 further includes executing a scene optimization process that iteratively applies proposed modifications to the AOIs, the distractor regions, and the NSRs, evaluates resultant saliency metrics from the VAM 204, and selects the modified scene that maximizes compliance with one or more user defined visual attention goals.

[0094] FIG. 4 illustrates a flowchart of an exemplary process 400 for automatically triggering execution of one or more segmentation models (e.g., the one or more segmentation models 202 shown in FIG. 2), according to an embodiment of the present disclosure. The process 400 provides a detailed view of the operation described in block 304 of FIG. 3. The process 400 is embodied as one or more algorithms implemented by the at least one processor 102 (shown in FIGS. 1 and 2) of the at least onecomputing device 101 (shown in FIGS. 1 and 2). Further, the process 400 may be stored in the at least one memory 110 (shown in FIGS. 1) as instructions executable by the at least one processor 102.

[0095] At block 402, the process 400 begins. The process 400 moves from block 402 to block 404. Referring to FIGS. 1-4, at block 404, the at least one processor 102 is configured to receive image data from a user via the one or more input devices 106.

[0096] The process 400 moves from block 404 to block 406. At block 406, the user begin creating an area of interest (AOI). The user may create the AOI by drawing a bounding box, a polygon, or another closed contour, or providing a pixel mask, which may be a contiguous set of pixels, from an external tool.

[0097] The process 400 moves from block 406 to block 408. Block 408 includes checking if one or more triggers are satisfied. The one or more triggers may include a time-based trigger, an eventbased trigger, or a combination trigger, i.e., a combination of time-based and event-based triggers. If any of the triggers (i.e., the time-based trigger, the event based-trigger, or the combination trigger) are satisfied at block 408, the process 400 moves to block 410, which includes automatically executing the one or more segmentation models on each AOI.

[0098] Block 408 includes a sub-operation 408A, which includes checking for satisfaction of the time-based trigger. For the time-based trigger, if the user begins creating a rectangular AOI, a timer is started that counts up from 0 and resets to 0 every time dimensions of the rectangular AOI (e.g., width, height, etc.) are changed or a global position of a centroid of the rectangle in pixel coordinates is changed. The trigger is satisfied once the timer exceeds a preset threshold duration (e.g., X seconds). In this non-limiting example, once the trigger is satisfied, the one or more segmentation models are applied to the area enclosed by the annotation rectangle, or to an area within a tolerance range that is slightly larger or smaller than the dimensions of the rectangle at block 410.

[0099] In other embodiments, if a polygon or a contour, such as a circle or a triangle with rounded edges, creates the AOI, a timer may be started that counts up from 0 and resets to 0 every time the dimensions of the AOI (e.g., maximum width, maximum height, perimeter length, etc.) change or the global position of the polygon / contour (e.g., minimum and maximum horizontal axis position, minimum and maximum vertical axis position, etc.) in pixel coordinates is changed. One the trigger is satisfied when the timer exceeds the preset threshold duration, the one or more segmentation models may be applied to the enclosed area or a slightly larger area at block 410. If a closed contour or mask (contiguous set of pixels) is provided to create the AOI, then the timer may be started that counts up from 0 and resets to 0 every time the closed contour or mask changes in size or shape or moves to a different global position. Once the timer reaches the preset threshold duration, the one or more segmentation models are applied to the enclosed area or a slightly larger area at block 410.

[0100] The block 408 further includes a sub-block 408B, which includes checking if the eventbased trigger is satisfied. The event-based trigger is satisfied if the user initiates a subsequent action distinct from modifying the current AOI, such as creating a new AOI or defining a visual attentiongoal, specifying constraints on scene modifications that can be explored, or providing other input(s) to a user interface. Upon satisfaction of the trigger-based event, the one or more segmentation models are executed on the AOI that was most recently modified by the user and has not yet been analyzed at block 410.

[0101] The block 408 further includes a sub-block 408C, which includes checking if the combination trigger is satisfied. For example, the event-based trigger may only be enabled after a certain amount of time elapses since the most recently created AOI was modified in any way (e.g., change size or shape, moved in global position, etc.). For all triggering mechanisms, embodiments of the present disclosure may create a flag for each AOI to indicate whether the one or more segmentation models have already been applied or not. This triggering process may repeat for each AOI that is created.

[0102] Block 410 includes automatically executing the one or more segmentation models on each AOI. The execution of the one or more segmentation model generates a pixel-level segmentation of a target entity. In some embodiments, the one or more segmentation models are applied to the area enclosed by the AOI or to an area within a tolerance range that is slightly larger or smaller than the dimensions of the AOI.

[0103] The process 400 moves from block 410 to block 412 after automatic execution of the one or more segmentation models. Block 412 includes presenting the segmentation results to the user, for example, via the one or more displays 104, in a format that identifies the initial AOIs.

[0104] Subsequently, the process 400 moves from block 412 to block 414. Block 414 includes checking for creation of a new AOI by the user. If creation of the new AOI is detected at block 414, the process 400 loops back to block 406. If creation of the new AOI is not detected at block 414, the process 400 moves to block 416 where the process 400 concludes.

[0105] FIGS. 5A-5B illustrate exemplary user interfaces 500, 501 for displaying automatic execution of the one or more segmentation models 202 (shown in FIG. 2), according to an embodiment of the present disclosure. Specifically, FIG. 5A illustrates the user interface 500 displaying an input image 502 (i.e., an original scene) provided by the user. FIG. 5B illustrates the user interface 501 displaying the same scene after the user has created one or more areas of interest (AOIs), including a first area of interest 504 (first AOI 504) and a second AOI 506. FIG. 5B further illustrates a first segmented boundary 508 and a second segmented boundary 510, which are automatically generated by the one or more segmentation models 202.

[0106] Referring to FIGS. 5A and 5B, a user draws the first AOI 504 and the second AOI 506 (e.g., rectangular boxes) over the input image 502 to indicate areas of interest. In some examples, the first AOI 504 and the second AOI 506 may be highlighted in a distinguishing color, e.g., green dashed lines. In response to a trigger condition being satisfied, the one or more segmentation models 202 are automatically executed on each AOI. The execution of the one or more segmentation models 202 may results in the generation of the first segmented boundary 508, which precisely delineates a first entitywithin the first AOI 504, and the second segmented boundary 510, which precisely delineates a second entity within the second AOI 506. In some examples, the first segmented boundary 508 and the second segmented boundary 510 may be highlighted in a distinguishing color, e.g., white and black dashed lines.

[0107] By automating this execution, a user experience of interacting with the one or more segmentation models 202 may be simplified, and rendering of the segmented entities may be expedited. The presentation of the first segmented boundary 508 and the second segmented boundary 510 allows the user to visually confirm an accuracy of the segmentation and, if necessary, provide additional inputs to refine the segmentation, as described in embodiments herein.

[0108] In an alternative embodiment, a two-stage trigger process may be executed for triggering the one or more segmentation models 202. In such cases, instead of executing the one or more segmentation models 202 immediately after an AOI is created by the user, the triggering process waits until all AOIs are defined by the user. A first stage of the two-stage trigger process, which may be functionally identical to the time-based and event-based triggers described in the process 400, is configured to trigger saving of boundary coordinates or enclosed pixels for each AOI. This approach allows any subsequent modifications to a created AOI to be quickly saved before the one or more segmentation models 202 are triggered. A second stage of the two-stage trigger process, which is functionally identical to the first stage, then executes the one or more segmentation models 202 on the image content within all the saved AOIs. While this sequence reduces a total number of queries sent to the one or more segmentation models 202, the user may not see the segmentation predictions immediately after creating each AOI.

[0109] In conjunction with automatically segmenting entities as described in operation 304, the at least one processor 102 may automatically classify each segmented entity, as described in operation 306. For example, multiple segmentation and classification models (e.g., the classification model 208) may be utilized, along with customized logic to manage, modify, and analyze the inputs and outputs from these models. In some embodiments, depending on the content captured within an AOI, only one segmentation or computer vision model may be utilized, while in other embodiments, several models may be applied to the same AOI. A class may be determined for each segmented entity based on the types of models that are applied.

[0110] For example, to automatically segment and classify an entity, the at least one processor 102 may predict a mask of pixels linked to the entity and predict a class label affiliated with that mask. In a specific embodiment for classifying text, one or more optical character recognition (OCR) models may be utilized to determine if text is present within a specified region and to identify the location of the text in pixel coordinates. If the OCR model finds text, a “text” class label is predicted. Subsequently, additional models may be utilzied, such as pre-trained segmentation models specifically tailored to text, to extract the precise pixel masks of every text character, including grammatical marks and numbers, found by the OCR models.[oni] FIGS. 6A-6C illustrate an exemplary comparison of different outputs of the one or more segmentation models 202 (shown in FIG. 2) displayed on user interfaces 600, 620, 630, according to an embodiment of the present disclosure. Specifically, FIG. 6A illustrates the user interface 600 displaying an output of an instance segmentation model applied to an input image 602. The output includes a plurality of bounding boxes 604, a plurality of class labels 606, and a plurality of precise pixel-level masks 608 that outline each detected entity. FIG. 6B illustrates the user interface 620 displaying an output of a semantic segmentation model applied to the input image 602. The output includes a plurality of semantic regions 610 where each pixel is classified into a category. FIG. 6C illustrates the user interface 630 displaying an output of a panoptic segmentation model applied to the input image 602. The output includes a plurality of panoptic masks 612 that assigns both a class label and a unique identifier to every pixel.

[0112] The at least one processor 102 may apply different types of segmentation models to provide varying levels of scene understanding. Referring to FIG. 6A, the instance segmentation model may be combined with an object detection model, detects multiple entities and provides the bounding boxes 604, the class labels 606, such as “bowl” or “bottle”, and the detailed pixel-level masks 608 for each entity. The ability of the instance segmentation model to distinguish between different instances of the same class label provides a high level of detail and accuracy regarding the exact location and shape of each entity. In some examples, each entity may be highlighted in a distinguishing color.

[0113] Referring to FIG. 6B, the semantic segmentation model may classify every pixel in the input image 602 into a predefined category, generating the plurality of semantic regions 610. The semantic segmentation model labels each semantic regions 610 with a category, such as “person”, “bottle”, and so on, effectively partitioning the image into meaningful segments. This approach is useful for quickly grasping an overall composition of a scene. However, the semantic segmentation model lacks the ability to distinguish between multiple instances of the same class, which can be a limitation when instance-level information is important. For example, additional refinements may be applied to the predictions from the semantic segmentation model if the user wants to focus on a specific article of clothing in a large wardrobe, or a specific item of food in a large display case at a bakery.

[0114] Referring to FIG. 6C, the panoptic segmentation model may combine the strengths of both instance and semantic segmentation. The panoptic segmentation model generates the plurality of panoptic masks 612, assigning a semantic label to every pixel while also uniquely identifying each instance of an entity. The panoptic segmentation model not only classifies each pixel into categories, such as “person”, “bowl”, or “apple,” but also uniquely identifies each entity instance. This holistic approach may provide a comprehensive understanding of a scene, providing both overall context and specific details about individual entities.

[0115] In some embodiments, for each segmented entity, the predicted mask label may be used identically as the class label. In another embodiment, the mask label may be used to predict a class label by employing traditional AI / ML models, such as a random forest classification model, a neuralnetwork classification model, or a pre-trained text analysis and semantic understanding model. Such models may be trained on text data to transform the mask label into the appropriate class label, or pretrained text analysis and semantic understanding models may be applied to infer a more general class label from a specific mask label. Such models may empower the user to more easily customize and control downstream processes. For example, if sky and clouds are segmented and classified, the user may apply constraints that only allow changes to brightness to those entities, thereby preserving a realistic appearance of the sky and clouds during a downstream optimization process.

[0116] FIGS. 7A and 7B illustrate exemplary user interfaces 700, 701 displaying automatic classification of segmented entities, according to an embodiment of the present disclosure. Specifically, FIG. 7A illustrates the user interface 700 displaying the classification of a segmented entity as an “object,” showing a first area of interest, AOI, 702, a first segmented boundary 704, an interactive toolbar 706, and a classification notification 708. FIG. 7B illustrates the user interface 701 displaying the classification of a segmented entity as “text,” showing a second AOI 710, a second segmented boundary 712, and an interactive toolbar 714.

[0117] Referring to FIGS. 7A and 7B, after a mask of pixels is segmented, the segmented entity may be classified and the class type is presented to the user for review. As shown in FIG. 7A, after a donut is segmented within the AOI 702, the segmented boundary 704 and the classification notification 708 is presented, which informs the user that the segmented entity has been recognized as an “object.” In some examples, the AOI 702 may be highlighted in a distinguishing color, e.g., green, and the segmented boundary 704 may be highlighted in a distinguishing color, e.g., black and white dashed lines.

[0118] Similarly, as shown in FIG. 7B, after the word “EXTREME” is segmented within the AOI 710, the segmented boundary 712 is presented and a text-related icon is highlighted on the interactive toolbar 714, indicating that the segmented entity has been classified as “text.” The user interfaces 700, 701 provide the interactive toolbars 706 and 714, respectively, to allow the user to review and, if necessary, manually change the class type.

[0119] The classification of a segmented entity into a class type, such as “object” or “text,” provides useful information for researchers and downstream processes. For developers of visual attention models (VAMs), knowing the class of content being analyzed is beneficial, as some entity classes may be more salient to specific types of visual attention neurons in the brain. For example, text is often more salient in visual scenes due to the presence of neurons that are only sensitive to intersection of edges that are oriented at specific angles. If developers of VAMs have more information about the content being analyzed, the developers could focus their finite resources on developing features that appeal to most users.

[0120] Furthermore, for a user seeking to modify a scene to achieve specific attention goals, the class type of an entity dictates permissible types of modifications that may be explored. If the modifiable entities in a scene are all objects or their shadows, then possible changes may be limited tocolor properties, such as hue, saturation, brightness, and contrast, with a limit of low brightness and saturation for the shadows. However, if a modifiable entity is classified as text, additional modifications, such as font changes, italicizing, underlining, and boldening may also be applied. Therefore, providing users with the class type for each entity may improve the performance of downstream processes.

[0121] FIGS. 8A and 8B illustrate exemplary user interfaces 800 displaying manual reclassification of a segmented entity, according to an embodiment of the present disclosure. Specifically, FIG. 8A shows the user interface 800 where a segmented entity has been automatically classified within an area of interest (A OI) 802. The user interface 800 includes a segmented boundary 804, an interactive toolbar 806, and a highlighted text icon 808, indicating the automatic classification of “text.” FIG. 8B shows the user interface 801 after the user has manually changed the classification to “object,” resulting in an updated segmented boundary 810, an updated interactive toolbar 812, and a highlighted object icon 814.

[0122] Referring to FIGS. 8A and 8B, once a class for each segmented entity is predicted, the user may either accept the predicted class or elect to change the classification manually. FIG. 8A shows a scenario where the entity “EXTREME” is automatically detected and classified into the class type as “text.” FIG. 8B shows a scenario where the user manually changes the classification type to “object.” This ability of the user to review and change the class type is useful because an entity may be incorrectly classified, or the entity is correctly classified but the user wants the segmented entity to be treated as another class type in the downstream processes.

[0123] In some embodiments, if the user changes the class type of an entity, the associated one or more segmentation models 202 (shown in FIG. 2) may be rerun on that AOI, as shown by the updated segmented boundary 810 in FIG. 8B. In an alternative embodiment, the user may change the class type of an entity, but the one or more segmentation models 202 are not rerun on that AOI, thereby preserving the original segmented boundary 804. The decision to rerun the one or more segmentation models 202 may be driven by user input or by an automated decision process executed by the at least one processor 102.

[0124] Furthermore, once the predicted segmented entities and the class types for the predicted segmented entities are returned, additional inputs may be accepted from the user to refine or change the segmented masks. The general progression includes the user first selecting the one or more segments and then providing new inputs to refine the selected segment or segments. Subsequent operations may include rerunning the one or more segmentation models 202, and finally, rendering the refined segment or segments for the user to review. The user may select a single segment by either drawing a bounding box or contour that fully encloses the segment, by clicking on a point within the single segment, or by providing another input that uniquely identifies the single segment.

[0125] In some embodiments, after the segment is selected, the user may refine the segment by providing additional inputs in the form of an inclusion point prompt or an exclusion point prompt. Theinclusion point prompt may include the user clicking on a point outside the segment that should be included in the segment. Flagging the point for inclusion may be performed by the user hitting a specific keystroke after clicking the point, or by creating a list menu displayed from the point where the user selects an “include” option. The exclusion point prompt may include the user clicking on a point inside the segment that should be excluded from the segment. Flagging the point for exclusion may be performed in a similar manner. One or multiple inclusion or exclusion point prompts may be provided for the segment.

[0126] In alternative embodiments, the user may refine a selected segment by providing an internal mask prompt or an external mask prompt. Regarding the internal mask prompt, some segmentation models may predict multiple masks for a single entity and return a best estimated mask. If more than one mask is predicted, the at least one processor 102 may present these masks through the user interface, and the user may select one or more of the masks to provide as an input. Regarding the external mask prompt, the user may interact with the system 100 through a software application that includes other tools. If a mask is generated by another software application, such as Adobe® Photoshop and tools, such as auto selection tool from Adobe®, then the user may provide that mask as an input.

[0127] FIG. 9 illustrates a flowchart of an exemplary process 900 for refining a segmented entity, according to an embodiment of the present disclosure. The process 900 provides a detailed view of the refinement operation, as described in embodiments herein. The process 900 is embodied as one or more algorithms implemented by the at least one processor 102 of the at least one computing device 101. Further, the process 900 may be stored in the at least one memory 110 as instructions executable by the at least one processor 102.

[0128] At block 902, the process 900 begins. The process 900 moves from block 902 to block 904. At block 904, one or more segmented entities are presented to the user. The process 900 then moves to block 906.

[0129] At block 906, a user selection of a single segmented entity is received for refinement. The process 900 then moves to block 908.

[0130] At block 908, the one or more user refinement inputs are received for the selected segmented entity. These inputs may include inclusion point prompts, exclusion point prompts, internal mask prompts, or external mask prompts. The process 900 then moves to decision block 910.

[0131] At the decision block 910, satisfaction of a trigger is checked to re-execute the onr or more segmentation models 202 (shown in FIG. 2). The block 910 may include a sub-block, which includes checking for a time-based trigger. For the time-based trigger, the at least one processor 102 initiates a timer when a new refinement input is provided, resets the timer upon receipt of each subsequent input, and determines if the trigger is satisfied when the timer exceeds a preset threshold duration. The block 910 may include a further sub-block, which includes checking for an event-based trigger. The eventbased trigger is satisfied when the user initiates a subsequent action, such as selecting another AOI,creating or modifying an attention goal, creating or modifying a modification constraint, or providing other input to the user interface not affiliated with the selected segment. If the trigger condition is not satisfied at block 910, the process 900 loops back to block 908 to await further refinement inputs. If the trigger condition is satisfied at block 910, the process 900 moves to block 912.

[0132] At block 912, the one or more segmentation models 202 are executed on the selected segment using the received refinement inputs. The process 900 then moves to block 914.

[0133] At block 914, the refined segmentation results are presented to the user. The process 900 then moves to decision block 916.

[0134] At the decision block 916, selection of another segmented entity by the user is checked. If the selection of another segment is detected at block 916, the process 900 loops back to block 906 to receive the selection. If the selection of another segment is not detected at block 916, the process 900 moves to block 918. At block 918, the process 900 concludes.

[0135] FIGS. 10A-10B and 11A-11B illustrate exemplary user interfaces 1000, 1001, 1100, 1101 showing refinement of segmented entities using user inputs, according to embodiments of the present disclosure. Specifically, FIGS. 10A and 10B illustrate the user interfaces 1000, 1001 where the user input create a new segmented entity linked with an original segment. FIGS. 11A and 11B illustrate the user interfaces 1100, 1101 where user inputs expand the scope of an existing segment.

[0136] FIG. 10A shows the user interface 1000 displaying an input image 1002. An area of interest (AOI) 1004 is defined by a user, and an initial segmented entity 1006 is automatically generated, which precisely outlines a notebook as shown in FIG. 10A. In some examples, the AOI 1004 may be highlighted in a distinguishing color, e.g., green. FIG. 10B shows the user interface 1001 after the user has provided a new point prompt input 1008, indicated by a cursor with a '+' sign over an adjacent entity. In response to the new point prompt input 1008, a new segmented entity 1010 is generated outlining the adjacent entity, such as a paper folder as shown in FIG. 10B.

[0137] FIG. 11A shows the user interface 1100 displaying an input image 1102. In this state, the user may select an entity 1103, such as a plant, by clicking on a point within the entity 1103. FIG. 11 B shows the user interface 1101 after the user has provided multiple point prompt inputs 1104, indicated by '+' signs, to refine the selection. In response, an expanded segmented entity 1106 that now includes previously missing parts of the plant, such as the leaves indicated by the point prompt inputs 1104.

[0138] Referring to FIGS. 9-1 IB, once the user provides the user inputs to refine or modify a specific segment, the execution of the one or more segmentation models 202 (shown in FIG. 2) on the segment to generate a refined segment may be triggered in multiple ways, including a time-based trigger, an event-based trigger, or a combination thereof, as described in the process 900. For the timebased trigger, once the user provides a new input to refine a selected segment, a timer may start that counts up from 0 and resets to 0 every time a new input is provided. Once the timer reaches a preset threshold duration, the one or more segmentation models 202 are triggered to process the segment with the additional user inputs.

[0139] For the event-based trigger, if the user selects (through clicking) or begins to select (drawing a closed contour) an AOI not affiliated with a segment or a specific segment in an AOI not affiliated with the segment, begins creating a new attention goal or modifying an existing goal, begins creating a new modification constraint or modifying an existing constraint, or provides other input to the user interface not affiliated with the segment, then the one or more segmentation models 202 are applied to the previously selected segment using the additional user inputs.

[0140] By allowing the user to refine segments through additional inputs, the system 100 may accurately segment entities in an enhanced manner required for downstream processes. For example, if the user wants to change a distribution of visual attention in a scene by modifying specific regions, the accurately segmented entities provide an environment where local values of hue, contrast, saturation, brightness, and other properties may be modified without creating artificial features in the surrounding regions. Similarly, if the user wants to remove an entity, a segment linked to that entity uniquely identifies the set of pixels to be filled in after the entity is erased.

[0141] In alternative embodiments, for refining segments, the user may select multiple segmented entities. For example, the user may select the multiple segments by drawing a bounding box or contour that fully encloses the multiple segments, or by clicking on multiple points where each point is within a different segment. Once a set of multiple segments is selected, the user may refine the segments through several options. In a first option, the user may provide an input to link distinct segments and treat the distinct segments as portions of the same broader entity. For example, the user may indicate a desire to link the segments by hitting or holding a specific keyboard key, such as “SHIFT key”, or by toggling a button in the user interface, and clicking once on each of the segments that should be linked. In this scenario, the one or more segmentation 202 models are not rerun.

[0142] In a second option, the user interface may render a 'MERGE' button that, upon a user click, instructs the system 100 to merge the selected segments into a single contiguous segment. In a third option, the user interface may render a “MERGE WITH REFINEMENT” button that, upon a user click, prompts the user to define inclusion and exclusion point prompts. Once the user is finished providing the point prompts, the system 100 is instructed to merge the selected segments into a single contiguous segment that respects the provided point prompts. For both merge options, the user may later be provided with an option to undo the merge and revert to a previous state with multiple distinct segments.

[0143] In a further embodiment, the user may create a segment from all parts of a scene that have not yet been segmented. To achieve this, the user interface may render a “SEGMENT OTHER” button. When the user clicks this button, the at least one processor 102 is configured to treat all pixels outside of existing segments as a single new segment, but does not assign a class type or attempt to predict one for this new segment. This can be useful in defining non-salient region (NSR) for subsequent modification.

[0144] In another embodiment, the at least one processor 102 may be configured to segment entities outside of areas identified by the user to provide previews of updated segments in real time or near real time. In this configuration, the one or more segmentation models 202 (shown in FIG. 2) are applied to regions that expand beyond the user-defined AOIs, potentially extending to the entire scene. After initial segments are returned and the user selects a segment to refine, an updated segment is highlighted in response to the user placing a cursor on a pixel near the selected segment but not currently included. This capability enables the user to more quickly refine segments, but at the cost of using more computer resources.

[0145] For segments that are classified as “text,” the user interface may render additional user input fields through which the user may specify what types of changes should be considered for the text in downstream processes, such as scene optimization. These changes may include different font types, such as Helvetica or Times New Roman, selected from a list, as well as changes to font size, font weight, italicization, underscore, shadow, bevel, emboss, gradient, text orientation, and other text characteristics. By default, all change modes may be disabled, and the user may select specific modes. For binary change modes like italicization, the user enables the desired change mode. For change modes that take discrete or continuous values, like font size or text orientation, the user may enable the mode and either accept a default range of values or provide a specific minimum or maximum value. For example, for text orientation, the user may accept a default range of rotation values, such as from 0 degrees to 180 degrees, or provide a desired range of orientation values.

[0146] For segments that the user flags as eligible for color changes, the changes may be constrained to specific values based on user input. Color change types, such as hue, saturation, and brightness, may be prohibited completely or restricted to specific continuous ranges defined by minimum and maximum values. For example, the user may constrain the hue of a segment to decrease by as much as 15 degrees or increase by as much as 10 degrees relative to its normal value. As another example, the brightness may be constrained to decrease by up to 20 percent or increase by up to 33 percent.

[0147] FIG. 12 illustrates an exemplary embodiment of a segmented entity that is constrained to specific hue values, according to an embodiment of the present disclosure. FIG. 12 shows a first version 1202 of a logo having a first hue value and a second version 1204 of the logo having a second, different hue value. For example, the first version 1202 may have a red hue, while the second version 1204 may have a black hue.

[0148] Embodiments of the present disclosure may allow users to define constraints whereby a color change is allowed, but the color change may only take specific values provided by the user. For example, a segment that represents a company logo, such as the logo shown in FIG. 12, may be changed from the first version 1202 to the second version 1204 only, where each version corresponds to a specific hue value that is permitted by the company. These specific values may be provided on demandby the user when the constraint is created, or the values may be provided by the user previously and saved to a user profile for easier access and persistence.

[0149] This functionality may be applicable in scenarios where advertisements must adhere to region-specific marketing rules. For example, physical products made by one company and sold in a first country and a second country may be distinguished by a specific set of colors for the first country and a completely different set of colors for the second country. By using specific value constraints, advertisements for these products in each respective country may be configured to only show the product colors available for sale in that specific country.

[0150] FIGS. 13A and 13B illustrate exemplary embodiments where a physical product is distinguished by specific color values based on production years, according to embodiments of the present disclosure. Specifically, FIG. 13A shows a first version 1302 of a bicycle corresponding to a first production year and FIG. 13B shows a second version 1304 of the same bicycle model corresponding to a second, different production year.

[0151] Referring to FIGS. 13A and 13B, the specific value constraints are applicable to scenarios where a physical product, made and sold in one country with the same name across multiple years, is distinguished by a new set of specific colors for each new production year. For example, the first version 1302 of the bicycle may be a model produced in 2023, while the second version 1304 of the bicycle may be the same model produced in 2021. In this embodiment, the color offered for the 2021 model is not available for the 2023 model, and vice versa. Similar types of specific value constraints may also be placed on other color change types, such as saturation and brightness, to allow users to precisely control how their content is modified.

[0152] FIG. 14 illustrates exemplary feedback systems for obtaining user preference feedback, according to an embodiment of the present disclosure. Specifically, FIG. 14 shows a star-rating system 1402 and a thumbs up rating system 1404. In some embodiments, the user may designate which of the multiple designs is preferred or may determine a level of acceptance for a single design through the feedback systems. For example, after a scene optimization process is run and results are returned, the system 100 may present the top-scoring designs in terms of meeting visual hierarchy goals of the user. The user may subsequently evaluate and provide feedback on these top-scoring designs.

[0153] As shown, the user may provide feedback in the form of a rating using the star-rating system 1402. Alternatively, the user may provide feedback using the thumbs up rating system 1404, which may include options to disapprove or approve a design. These user-provided preferences or ratings could subsequently be passed back into a scene optimization process or other downstream processes as inputs. While two examples of feedback systems are shown, many other ways for obtaining user preference feedback are possible.

[0154] FIG. 15 illustrates an exemplary user interface 1500 displaying application of different modification constraints and visual attention goals to multiple areas of interest, according to an embodiment of the present disclosure. Specifically, FIG. 15 shows the user interface 1500 displayingan input image 1502, a first area of interest (A 01) 1504, a second AOI 1506, a third AOI 1508, and a segment 1510 within the first AOI 1504.

[0155] In embodiments where two or more AOIs are defined or two or more segments are created, the system 100 allows the user to select two or more of the AOIs or segments and apply the same modification constraints or define a single visual attention goal that encompasses all selected regions. As shown in the exemplary scene, the user defined the first AOI 1504, the second AOI 1506, and the third AOI 1508. The system 100 also may have identified one or more of these as distractor regions. The user may select the first AOI 1504, which contains the segment 1510, and specify that a downstream optimization process should find ways to increase an attention score of the first AOI 1504 by modifying the hue, saturation, and brightness of the enclosed content. In some examples, the segment 1510 may be highlighted in a distinguishing color, e.g., green. Similarly, the AOIs may be highlighted in a distinguishing color.

[0156] Subsequently, the user may select the second AOI 1506 and specify that no changes should be made. Finally, the user may select the third AOI 1508 and specify that the attention score for the third AOI 1508 should be reduced by modifying the hue, saturation, and brightness of content within the third AOI 1508. These goals and constraints matriculate automatically to the segments enclosed within each respective AOI. As shown, a single AOI, such as the first AOI 1504, may correspond to one or multiple segments. This capability enables users to work faster and more efficiently when providing inputs that are redundant across multiple regions.

[0157] FIG. 16 illustrates an exemplary user interface 1600 displaying a user-defined gaze sequence goal that spans multiple areas of interest (AOI), according to an embodiment of the present disclosure. Specifically, FIG. 16 shows the user interface 1600 displaying an input image 1602, a first area of interest (AOI) 1604, a second AOI 1606, a first gaze sequence input 1608, and a second gaze sequence input 1610.

[0158] The ability to select multiple AOIs or segments enables the user to define gaze sequence goals that encompass two or more regions. In the exemplary scene shown, the user defined the first AOI 1604 and the second AOI 1606. The user further defines a gaze sequence goal by providing the first gaze sequence input 1608, represented by a numbered circle, and the second gaze sequence input 1610, also represented by a numbered circle. In this non-limiting example, the user may wish a first predicted gaze sequence point to fall somewhere inside the first AOI 1604, and a second predicted gaze sequence point to fall somewhere inside the second AOI 1606. In some embodiments, the user may instead require that one or both gaze sequence points fall inside the specific segment within each respective AOI. Through the user interface 1600, the user may specify this goal, for example, by toggling a button to indicate that a gaze sequence goal is being created, clicking on each AOI or segment in the order of the desired gaze sequence, and then un-toggling the gaze sequence mode button. The at least one processor 102 may populate a gaze sequence goal using these inputs for a downstream optimization process.

[0159] FIG. 17 illustrates an exemplary user interface 1700 displaying a user-defined gaze sequence goal that spans multiple areas of interest without a specified order, according to an embodiment of the present disclosure. Specifically, FIG. 17 shows the user interface 1700 displaying an input image 1702, a first area of interest (A OI) 1704, a second AOI 1706, a third AOI 1708, and a fourth AOI 1710.

[0160] The ability to select multiple AOIs or segments may also simplify defining a gaze sequence goal that does not include a specific gaze sequence order. As shown in the exemplary scene, the user selected the first AOI 1704, the second AOI 1706, the third AOI 1708, and the fourth AOI 1710. If the user wishes the first through the fourth predicted gaze sequence points to fall somewhere within these four AOIs but does not require the gaze points in a specific order, the user may select all four AOIs and specify this collective goal. The selection may be made, for example, by clicking each AOI individually while holding a specific key, or by using a drag-and-draw function to create a box that encloses all four AOIs. Notably, without specifying an order, a gaze sequence goal such as this example may be satisfied without requiring one gaze sequence point to fall in each region.

[0161] In some embodiments, if one or more gaze sequence goals are defined but cannot be achieved by modified scenes produced by scene optimization processes, a best modified scene or a set of best modified scenes as alternative solutions may be recommended. Although none of these alternative modified scenes meet all the gaze sequence goals, providing the alternative modified scenes to the user may help the user to identify what content in the scene should change or what is preventing the goals from being met. Algorithms such as simulated annealing optimization, greedy optimization, and weighted scoring systems may be used to determine which modified scenes come closest to achieving the gaze sequence goals.

[0162] FIG. 18 illustrates an exemplary user interface 1800 displaying output of a scene optimization process, according to an embodiment of the present disclosure. The user interface 1800 provides a detailed breakdown of modifications made to an input image 1802 and their effect on visual attention metrics. The user interface 1800 includes a main toolbar 1804 for navigating different analysis types, a display of the modified input image 1802 with overlaid areas of interest (AOIs), and a results panel 1806 that quantifies changes for each AOI.

[0163] After a scene optimization is performed by the one or more optimization models 206 (shown in FIG. 2), the at least one processor 102 may output the user interface 1800 on the one or more displays 104. The results panel 1806 provides a comprehensive summary for each defined AOI, such as a first AOI 1808 (labeled “A”), a second AOI 1810 (labeled “B”), and a third AOI 1812 (labeled “C”). For the first AOI 1808, the results panel 1806 displays an attention score display 1816, showing both a final attention percentage and a percentage increase. The results panel 1806 further provides modification details 1818, specifying that changes to hue and saturation were applied to achieve this result. For the second AOI 1810, the results panel 1806 includes a protection indicator 1820, which communicates to the user that this AOI was marked as protected and therefore no modifications wereapplied during the optimization process. For the third AOI 1812, the results panel 1806 shows a decrease in the attention score and details the corresponding change in a saturation parameter. This optimization may also include modifications to identified distractor regions or non-salient regions (not explicitly shown in panel 1806) which contributed to changes in attention scores for AOIs.

[0164] In some embodiments, when a scene optimization process is run, the user is enabled to download the optimization results to a local machine. For example, the user interface 1800 includes a download feature 1814, which may be represented by an icon and / or text. Upon user activation of the download feature 1814, data associated with the optimization may be compiled and downloaded. This downloaded data may include, but is not limited to, the modified input image 1802, a data file containing the final attention scores for each AOI (i.e., the first AOI 1808, the second AOI 1810, and the third AOI 1812), a summary of the specific visual property modifications (e.g., hue, saturation, brightness values) applied to each segmented entity, and any user-defined goals or constraints. This functionality provides the user with a tangible record of the optimization process, suitable for archiving, sharing with collaborators, or for use in subsequent design and marketing workflows.

[0165] FIGS. 19A and 19B illustrate exemplary user interfaces 1900, 1901 showing a recommended scene modification from an optimization process, according to an embodiment of the present disclosure. Specifically, FIG. 19A illustrates an original scene 1902 prior to optimization. FIG. 19B illustrates a modified scene 1904 after the optimization process has been applied. The modified image 1904 shown in FIG. 19B includes an area of interest (AOI) 1906, which has been modified by the one or more optimization models 206 to achieve a specific visual attention goal. In this example, the goal is to maximize the attention score for the AOI 1906, resulting in a displayed attention score 1908 of 50%.

[0166] In some embodiments, the system 100 may estimate a sensitivity of saliency metrics, which is implicit in the visual attention goals, to changes in scene modifications. After the system 100 provides a recommendation, such as the one shown in the modified scene 1904, the user may accept the recommendation and implement the corresponding changes in a separate program, or the user may instruct the system 100 to perform a sensitivity analysis. If the sensitivity analysis is requested, the at least one processor 102 is configured to explore variations in scene modification parameters applied to the AOI 1906 and determine an impact of the variations on the associated saliency metrics. A user may also define new visual attention goals based on the sensitivity metrics and re-run the scene optimization.

[0167] Table 1 below illustrates an exemplary output of such a sensitivity analysis, presented in a tabular format. The analysis explores effect of changes in hue, brightness, and saturation on an attention score of the AOI 1906. The first row of Table 1 represents the recommended values from the optimization process, where a change of 0 degrees for hue, 0 percent for brightness, and 0 percent for saturation results in the maximized attention score of 50.0. This score corresponds to the attention score 1908 shown in FIG. 19B.Table 1

[0168] Referring to Table 1, the subsequent rows show estimated attention scores resulting from variations in hue, brightness, and saturation relative to recommended set of values. For example, a hue change of -3.0 degrees, a brightness change of -3.0 percent, and a saturation change of 3.0 percent are estimated to decrease the attention score to 42.5. The results presented in Table 1 indicate that even small deviations in hue, brightness, and saturation from the recommended values are likely to significantly decrease the attention score for the AOI 1906. This sensitivity analysis provides the user with valuable insight into the robustness of the optimization and the criticality of each visual parameter for achieving the desired attention goal. Although the present example utilizes an attention score as the saliency metric, in other embodiments, the saliency metric may include gaze sequence position and other representations of visual attention.

[0169] FIGS. 20A and 20B illustrate exemplary user interfaces 2000, 2001 displaying a visual representation of a result from the sensitivity analysis described in reference to Table 1. Specifically, FIG. 20A depicts an original scene 2002, which is the same as the original scene 1902 shown in FIG.19A. FIG. 20B depicts a modified scene 2004 where a content within an area of interest (AOI) 2006 has been altered according to a specific variation from the sensitivity analysis.

[0170] Referring to FIG. 20B, the modified scene 2004 represents a state where the content within the AOI 2006 is modified such that the hue is reduced by approximately 3 degrees, the brightness is increased by approximately 3 percent, and the saturation is increased by approximately 3 percent, relative to the recommended scene modification shown in FIG. 19B. These specific parameter changes correspond to the final row of data presented in Table 1. As a result of these modifications, the at least one processor 102 estimates and presents a displayed attention score 2008 of 42%. This visualization allows a user to directly observe the visual impact of deviating from the optimal recommended parameters and see the corresponding degradation in the attention score, from 50.0 to approximately 42%.

[0171] FIGS. 21A and 21B illustrate exemplary user interfaces 2100, 2101 displaying a new recommended scene modification resulting from a re-executed optimization process, according to an embodiment of the present disclosure. Specifically, FIG. 21A depicts an original scene 2102, which is the same as the original scene 1902 shown in FIG. 19A. FIG. 21B depicts a modified scene 2104 where an area of interest (AOI) 2106 has been altered based on a new optimization goal.

[0172] Based on significant sensitivity of the attention score demonstrated in the analysis of Table 1, the user may prompt the system 100 to re-run the scene optimization. In this embodiment, the usermay define a new goal to not only maximize the attention score but also to minimize the variability of the attention score in response to changes in parameters such as hue, brightness, and saturation. The at least one processor 102 is configured to execute the one or more optimization models 206 (shown in FIG. 2) with this new dual objective. The system 100 may return one or multiple recommendations that satisfy this goal. As shown in the modified scene 2104, a specific recommendation may be to change the hue of the AOI 2106, causing the associated visual content to appear green. This modification results in a displayed attention score 2108 of 49%.

[0173] Table 2 below illustrates the sensitivity analysis for this new recommendation. The sensitivity analysis shows variations in hue, brightness, and saturation and their effects on the attention score of the AOI 2106. The modifications presented in Table 2 correspond to variations around the recommended scene shown in FIG. 2 IB.Table 2

[0174] Referring to Table 2, in addition to achieving a high attention score, the recommended modification also improves a robustness of the attention score to changes in hue, brightness, and saturation. For example, a hue change of 43.0 degrees, a brightness change of 10.0 percent, and a saturation change of 3.0 percent are estimated to produce an attention score of 54.0. The data demonstrates that the attention scores remain relatively high even with significant variations in the modification parameters, indicating that the new optimization has successfully identified a more stable solution compared to the previous analysis.

[0175] FIGS. 22A and 22B illustrate exemplary user interfaces 2200, 2201 displaying another visual representation of a result from the sensitivity analysis described in reference to Table 2. Specifically, FIG. 22A shows an original scene 2202, which is the same as the original scene 2102 shown in FIG. 21 A. FIG. 22B shows a modified scene 2204 where a content within an area of interest (AOI) 2206 has been altered according to another specific variation from the sensitivity analysis.

[0176] Referring to FIG. 22B, the modified scene 2204 represents a state where the brightness of the content within the AOI 2206 is significantly reduced. These specific parameter changes correspond to the third row of data presented in Table 2, which includes a hue change of 45.0 degrees, a brightness change of -30.0 percent, and a saturation change of -3.0 percent. As a result of these modifications, the at least one processor 102 estimates and presents a displayed attention score 2208 of approximately 46%. This visualization demonstrates that even with a significant reduction in brightness, the attentionscore does not decrease substantially, further highlighting the robustness of the optimized solution identified in the re-executed optimization process.

[0177] In some embodiments, however a scene optimization is performed, the sensitivity analysis may be conducted algorithmically. For example, the sensitivity analysis may be performed by characterizing all the modified scenes tested during the optimization, or by generating additional modified scenes that are similar to one or more recommended optimal scenes and applying the VAM 204 on those additional scenes. The characterization of tested scenes may be performed quickly with small computing resources, whereas the approach of generating and analyzing additional scenes may be more time consuming and require more computing resources due to re-running the VAM 204 multiple times. The latter approach may be necessary if a recommended modified scene is isolated in a phase space of possible scene modifications that were explored during optimization.

[0178] FIGS. 23A-23C illustrate exemplary user interfaces 2300, 2320, 2330 displaying aprocess of segmenting and refining entities within a scene, according to an embodiment of the present disclosure. In some embodiments, once the user uploads and selects an image to analyze, the user may draw areas of interest (AOIs) where more information is returned by a visual attention analysis (VAS) and may engage additional systems that segment and modify the content in those areas.

[0179] Referring to FIGS. 23A and 23B, engaging the one or more segmentation models 202 (shown in FIG. 2) enables the system 100 to identify and extract the pixels associated with specific entities, resulting in a first segmented entity 2304 corresponding to a text and a second segmented entity 2306 corresponding to a folder. The user may provide an additional input, such as an inclusion point prompt represented by a user input 2308 (shown in FIG. 23B), to further refine existing segments. Referring to FIG. 23C, a result of the refinement process is shown, where the text has been processed into a first refined segment 2310, and the second segmented entity 2306 (shown in FIG. 23B) has been expanded via the user input to form a second refined segment 2312, which now includes an overlapping folder on the right.

[0180] FIGS. 24A and 24B illustrate exemplary user interfaces 2400, 2401 for displaying different layers of a VAS analysis, according to an embodiment of the present disclosure. A user may select different analysis tools from a main toolbar to view different insights. Referring to FIG. 24A, the user interface 2400 is shown in a first state, displaying an “Areas of Interest” 2402 analysis view. In this view, visual attention metrics, such as percentage scores, are displayed for each defined AOI after segmentation and VAS analyses are complete.

[0181] Referring to FIG. 24B, the user interface 2401 is shown in a second state after the user has selected a gaze sequence tool 2404. This view displays a gaze sequence overlay 2406, which includes a plurality of numbered gaze points, such as a first gaze point 2408 (a stack of notes) and a second gaze point 2410 (a pen holder), predicting the path of attention of the viewer. In this embodiment, the at least one processor 102 identifies distracting elements as entities that are outside the primary AOIs butare highly salient, as indicated by an overlap with the gaze points. Accordingly, a first distractor 2412 (a stack of notes) and a second distractor 2414 (a pen holder) are identified and highlighted.

[0182] FIGS. 25A-25C illustrate exemplary user interfaces 2500, 2530, 2540 for specifying attention goals and modification constraints for segmented entities, including AOIs, distractor regions, and NSRs, according to an embodiment of the present disclosure. If the user wants to modify a scene to change the attention metrics of specific entities, the user may iteratively select each entity and specify a desired change and the types of modifications that may be made to achieve the attention changes.

[0183] Referring to FIG. 25A, the user may iteratively select each entity to change and specify a direction (increase, decrease, protect from modification) of change in an attention metric. For example, a first segmented entity 2502, corresponding to a text, is selected. An interactive toolbar 2504 appears, allowing the user to set an increase attention goal 2506 and to specify that this goal should be achieved by modifying a brightness constraint 2514. The interactive toolbar 2504 also includes options for a decrease attention goal 2508, a protection from modification goal 2510, a color constraint 2512, and a contrast constraint 2516.

[0184] Referring to FIGS. 25B and 25C, the user may continue to specify goals for other entities. As shown in FIG. 25B, a second segmented entity 2518, corresponding to a stack of notes (identified as a distractor region), is selected, and the user applies the protect from modification goal 2510. As shown in FIG. 25C, athird segmented entity 2520 (also identified as a distractor region), corresponding to a pen holder, is selected, and the user applies the decrease attention goal 2508 and specifies that the change should be achieved by modifying the contrast constraint 2516. Once the user specifies the desired attention changes and permissible modification types, the user may launch the optimization functions of the system 100.

[0185] FIGS. 26A and 26B illustrate exemplary user interfaces 2600, 2601 for displaying results of an optimization process, according to an embodiment of the present disclosure. After the optimization is complete, the system 100 may return one or multiple modified scenes that achieve the desired changes in attention metrics using different combinations of scene modifications.

[0186] Referring to FIG. 26A, the user interface 2600 displays an optimization summary view 2602. The optimization summary view 2602 includes a design insights panel 2604, which provides a summary of attention metric changes, and a design updates panel 2606, which details specific modifications made to each region (AOI, distractor region, or NSR) to achieve goals of the user. Referring to FIG. 26B, the user interface 2601 displays a comparison view 2608. The comparison view 2608 presents multiple result scenes, such as a first result scene 2610 and a second result scene 2612, in a grid format. Each result scene displays different attention metrics, allowing the user to directly compare the outcomes of different optimization strategies. For example, modifying a color content of a text and a pen holder may drive a text attention score higher while lowering an attentionscore of the pen holder. In another example, removing the pen holder using a generative fill function may drive the text attention score even higher.

[0187] FIG. 27 illustrates a flowchart of an exemplary process 2700 for optimizing a scene based on user-defined goals, according to an embodiment of the present disclosure. The process 2700 provides a detailed view of a holistic scene optimization operation. The process 2700 is embodied as one or more algorithms implemented by the at least one processor 102 of the at least one computing device 101 (shown in FIG. 1). Further, the process 2700 may be stored in the at least one memory 110 (shown in FIG. 1) as instructions executable by the at least one processor 102.

[0188] At block 2702, the process 2700 begins. The process 2700 moves from the block 2702 to block 2704. At block 2704, an image of a scene or image components that can create a scene is received from a user. The process 2700 then moves to an optional block 2706, where one or more object segmentation models (e.g., the one or more segmentation models 202) are executed on the image.

[0189] The process 2700 then receives a plurality of user inputs at blocks 2708, 2710, 2712, 2714. At block 2708, user-specified visual attention goals are received. At block 2710, user identified regions in the scene connected to the visual attention goals are received. At block 2712, user-specified constraints on modifications that can be made are received. At block 2714, a user-specified amount of time in which they can wait before scene change recommendations are needed is received (for the optimization process).

[0190] The process 2700 moves from blocks 2708, 2710, 2712, 2714 to block 2716. At block 2716, all user inputs are collected and transmitted to the system 100. The process 2700 then moves to block 2718. At block 2718, a visual attention model (e.g., the VAM 204) is executed on the image. The process 2700 then moves to block 2720. At block 2720, a VAM analysis is performed. The block 2720 includes two sub-blocks, block 2720A and block 2720B. At block 2720A, the VAM analysis includes generating predicted saliency or predicted fixations for AOIs and any distractor regions. At block 2720B, the VAM analysis further includes generating feature maps for the AOIs, any distractor regions, and NSRs.

[0191] The process 2700 moves from block 2720 to decision block 2722. At decision block 2722, the at least one processor 102 determines if all user goals are met by the initial VAM analysis. If all the user goals are met, the process 2700 moves to block 2724, where the image and VAM results are returned to the user, and the process 2700 concludes. If all the user goals are not met, the process 2700 moves to optional block 2726. At optional block 2726, the user is allowed to specify preferred content modification regions, such as a preference for editing an AOI, a distractor region, or an NSR.

[0192] The process 2700 moves from block 2726 to block 2728. At block 2728, one iteration of the optimization process is executed, which includes testing a combination of scene modifications and evaluating an effect of the scene modifications on the user goals. The process 2700 then moves to decision block 2732. At block 2732, the at least one processor 102 determines if the optimization loop should terminate. This determination is based on evaluating if either of the two conditions is satisfied:first, whether all user goals have been met by the latest tested modification, or second, whether the user-specified time limit for the optimization has been reached.

[0193] If neither condition is met, the process 2700 loops back to block 2728 via block 2730 to begin another optimization iteration. At block 2730, the at least one processor 102 estimates what modifications should be tested next based on previously tested modifications, change constraints, JND step increments, and user goals that have or have not been met. The output of operation 2730 provides input for the next iteration of operation at block 2728.

[0194] If either condition is met, the process 2700 moves to block 2734 to output the results. At block 2734, the at least one processor 102 is configured to output a recommended set of scene modifications and / or a modified image that achieves the user goals.

[0195] The process 2700 moves from block 2734 to block 2736. At block 2736, the system 100 translates the recommendations into instructions and contextual explanations. The process 2700 then moves to block 2738. At block 2738, instructions to apply the modifications are transmitted to the user via a graphical user interface. The process 2700 then moves to block 2740, where the process 2700 concludes.

[0196] FIGS. 28A-28C illustrate exemplary user interfaces 2800, 2810, 2820 describing generation of an attention-weighted image from an original scene, according to an embodiment of the present disclosure. Specifically, FIG. 28A shows an original image 2802 provided to the system 100 as input. FIG. 28B shows a predicted visual attention heatmap 2804, generated by the VAM 204, overlaid on the original image 2802. FIG. 28C shows a convolved image 2806, which is the result of combining the original image 2802 with the attention heatmap 2804.

[0197] Referring to FIGS. 28A-28C, the identification of distractor regions may be achieved through methods that build upon predictions made by the visual attention and object segmentation models. The VAM 204 is configured to predict where the attention will go across an entire scene and produce the attention heatmap 2804, as shown in FIG. 28B. To create the convolved image 2806 shown in FIG. 28C, the values from the attention heatmap 2804 are transformed to a [0, 1] scale and then multiplied with individual pixel values of the original image 2802. This process produces a convolved image that effectively weighs a content of the original image by the predicted attention.

[0198] In some embodiments, the transformation of the heatmap values to the [0, 1] scale may be performed in a linear manner, for example, by dividing each value in the heatmap 2804 by a maximum value found within the heatmap 2804. In some embodiments, the transformation may be performed in a non-linear manner, for example, by passing a pixel-level attention values to a sigmoid function. An inflection point of the sigmoid function may be set based on the visual attention goals of the user, allowing for a more customized weighting of the attention values.

[0199] FIGS. 29A and 29B illustrate exemplary user interfaces 2900, 2901 describing segmentation of visually salient objects, according to an embodiment of the present disclosure.Specifically, FIG. 29A shows a convolved image 2902. FIG. 29B shows a plurality of segmented objects 2904 identified from the convolved image 2902.

[0200] Referring to FIGS. 29A and 29B, the convolved image 2902 may be provided to an object segmentation algorithm. The segmentation algorithm is configured to process the convolved image 2902 to segment objects in the scene that are determined to be visually salient. This process results in generation of the plurality of segmented objects 2904, as shown in FIG. 29B, which represent the entities within the scene having high predicted visual attention. In some examples, the plurality of segmented objects 2904 may be highlighted in distinguishing colors.

[0201] FIGS. 30A and 30B illustrate exemplary user interfaces 3000, 3001 describing an alternative process for segmenting visually salient objects using a saliency threshold, according to an embodiment of the present disclosure. Specifically, FIG. 30A shows a threshold image 3002 generated from an original scene. FIG. 30B shows a plurality of segmented salient objects 3004 identified from the threshold image 3002.

[0202] Referring to FIGS. 30A and 30B, in an alternative method for identifying distractor regions, the original image may be filtered to display only those pixels with a saliency value above a certain threshold, resulting in the threshold image 3002. For example, as shown in the generation of FIG. 30B, each pixel may be required to have a saliency value greater than or equal to 40% of a maximum possible saliency value. This resulting scene is significantly simpler and may enable faster identification of distractor regions when the threshold image 3002 is provided to an object segmentation model. The object segmentation model may then generate the plurality of segmented salient objects 3004.

[0203] In some embodiments, the threshold may be a single value T, or may be defined by bounds T rnin and T max. The threshold or bounds may be calculated based on the predicted pixel saliency values across the entire scene or a subset thereof. The calculation may optionally incorporate user-defined numeric attention goals, predicted gaze sequence coordinate locations, or other visual attention model predictions. By applying this approach to the scene shown in FIGS. 28A-28C, 29A-29B, and / or 30A-30B, the system 100 may identify specific entities, such as a flower plant, a coffee grinder, and a stack of plates as highly attentive objects, and therefore as potential distractors, even though they are not related to the primary subject matter of the website.

[0204] FIGS. 31A and 31B illustrate exemplary user interfaces 3100, 3101 describing identification of distractor regions using predicted attention hotspots, according to an embodiment of the present disclosure. Specifically, FIG. 31A shows an original image 3102. FIG. 3 IB shows the original image 3102 with a plurality of attention hotspots overlaid, including a first hotspot 3106, a second hotspot 3108, and a third hotspot 3110. In some examples, the first hotspot 3106, the second hotspot 3108, and the third hotspot 3110 may be highlighted with distinguishing colors. For example, the first hotspot 3106 may be highlighted with a red color, the second hotspot 3108 may be highlighted with a yellow color, and the third attention hotspot 3110 may be highlighted with blue color.

[0205] Referring to FIGS. 31A and 3 IB, the distractor regions may be identified or further refined by checking for overlap between segmented objects and predicted gaze sequence points or the plurality of attention hotspots. For any gaze sequence point or portion of an attention hotspot that does not fall within a user-defined area of interest, the at least one processor 102 is configured to perform further analysis.

[0206] In some embodiments, this analysis includes two considerations. First, the overlap of a segmented object with a predicted gaze sequence point may provide evidence that the segmented object is a distractor. Comparisons between the gaze sequence point, the segmented object shape and size, and the entire image size may be made to better determine if the segmented object is a distractor. For example, if a distance between the gaze sequence point and a centroid of the segmented object is more than 10% of the larger image dimension, the segmented object is determined to be less likely to be a distractor. Second, the overlap of a segmented object with a portion of an attention hotspot may provide additional evidence that the segmented object is a distractor. Comparisons between shapes and sizes of the attention hotspot and the segmented object may be made to assess if the segmented object is a distractor. For example, if an area of overlap between the attention hotspot and the segmented object is more than a first percentage of an area of the segmented object and more than a second percentage of an area of the attention hotspot, then the segmented object is determined to be more likely to be a distractor.

[0207] As an exemplary application shown in FIG. 3 IB, the only user-defined area of interest is the region containing the text “#I RECOMMENDED BRAND” written in a lower right comer. If segmented objects in the scene are required to overlap with the third attention hotspot 3110 (having an 84% attention value), and an additional rule requires that at least 70% of an area of the segmented objects must fall within the third attention hotspot 3110, then an object in the image 3102 corresponding to a coffee grinder is identified as a distractor.

[0208] FIGS. 32A-32C illustrate a series of exemplary images and associated visual attention analysis (VAS) results, demonstrating effects of modifying different regions within a scene, including areas of interest (AOIs), distractor regions, and non-salient regions (NSRs). Specifically, FIG. 32A illustrates an original image 3202 and its associated VAS analysis 3204 and a VAS heatmap 3206. FIG. 32B illustrates an AOI edit image 3208 and its associated VAS analysis 3210 and a VAS heatmap 3212. FIG. 32C illustrates a distractor region edits image 3214 and its associated VAS analysis 3216 and a VAS heatmap 3218.

[0209] Referring to FIG. 32A, the original image 3202 contains multiple instances of the words “TARGET”, “DISTRACTOR,” and “Non-Salient Region”. The associated VAS analysis 3204 shows several user-designated AOIs, which correspond to the “TARGET” text instances, and identifies the “DISTRACTOR” text instances as distractors. The analysis displays initial attention scores for the AOIs, for example, 98% for AOI A, 35% for AOI B, and 76% for AOI C. The VAS heatmap 3206visually represents the predicted attention distribution, showing high saliency over both the “TARGET” and “DISTRACTOR” text instances.

[0210] Referring to FIG. 32B, the AOI edit image 3208 shows a version of the scene where the “TARGET” text instances have been modified, for example, by changing their color. The associated VAS analysis 3210 shows the resulting change in attention scores for the modified AOIs, for example, to 98% for AOI A, 76% for AOI B, and 75% for AOI C. The corresponding VAS heatmap 3212 confirms this shift in attention, with more intense hotspots now concentrated on the modified “TARGET” instances.

[0211] Referring to FIG. 32C, the distractor region edits image 3214 shows a version of the scene where the “DISTRACTOR” text instances have been modified, for example, by reducing their size and altering their color. The associated VAS analysis 3216 demonstrates the effectiveness of this change by showing the resulting impact on the AOIs. For example, attention scores for the “TARGET” AOIs are now 98% for AOI A, 49% for AOI B, and 94% for AOI C. The VAS heatmap 3218 visually confirms that attention has been drawn away from the modified distractor regions and redistributed across the scene.

[0212] FIG. 33A illustrates an NSR edit image 3302 and an associated VAS analysis 3304 and a VAS heatmap 3306, according to an embodiment of the present disclosure. The NSR edit image 3302 shows a version of the scene where the text corresponding to the non-salient regions has been modified, for example, by changing the color of the text to more closely match the background. The associated VAS analysis 3304 shows the resulting attention scores for the AOIs are now 98% for AOI A, 86% for AOI B, and 97% for AOI C. The corresponding VAS heatmap 3306 visually confirms that reducing the saliency of the NSRs causes attention to be more strongly consolidated on the remaining salient elements, namely the “TARGET” and “DISTRACTOR” text instances.

[0213] FIG. 33B illustrates a combined AOI, Distractor, and NSR edit image 3308, an associated VAS analysis 3310, and a VAS heatmap 3312, according to an embodiment of the present disclosure. Referring to FIG. 33B, the edit image 3308 shows a version of the scene where all three region types have been modified to achieve user goals. In this example, the “TARGET” text instances have been made more prominent, while the “DISTRACTOR” and “Non-Salient Region” text instances have been de-emphasized. The associated VAS analysis 3310 shows a highly successful optimization, with the attention scores for the AOIs now at 98% for AOI A, 96% for AOI B, and 98% for AOI C. The VAS heatmap 3312 visually represents this outcome, showing that the predicted attention is now almost exclusively concentrated on the “TARGET” AOIs.

[0214] FIG. 34 illustrates an exemplary user interface 3400 describing application of scaleinvariant feature transform (SIFT) algorithm for detecting and locating similar image regions, according to an embodiment of the present disclosure. The figure shows a reference region 3402, a larger search image 3404, and a plurality of matching keypoint lines 3406.

[0215] As shown in FIG. 34, the reference region 3402, which contains a brand logo, serves as a template. The at least one processor 102 is configured to apply a feature detection algorithm, such as the SIFT algorithm, to analyze the reference region 3402 and the larger search image 3404. The algorithm identifies unique keypoints in both images. The plurality of matching keypoint lines 3406 represent successful matches found between the reference region 3402 and multiple locations within the larger search image 3404. This process enables the system 100 to automatically identify all instances of the logo, allowing for synchronized editing across all linked regions.

[0216] In the present detailed description of embodiments, reference is made to the accompanying drawings, which illustrate specific embodiments in which the invention may be practiced. The illustrated embodiments are not intended to be exhaustive of all embodiments according to the invention. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

[0217] Unless otherwise indicated, all numbers expressing feature sizes, amounts, and physical properties used in the specification and claims are to be understood as being modified by the term “about”. Accordingly, unless indicated to the contrary, the numerical parameters set forth in the foregoing specification and attached claims are approximations that can vary depending upon the desired properties sought to be obtained by those skilled in the art utilizing the teachings disclosed herein.

[0218] Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and / or equivalent implementations can be substituted for the specific embodiments shown and described without departing from the scope of the present disclosure . This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that this disclosure be limited only by the claims and the equivalents thereof.

Claims

What is claimed is:

1. At least one non-transitory computer-readable medium storing instructions that, when executed, configure at least one processor forreceiving image data representing a scene;predicting regions associated with visual attention using a visual attention model (VAM) and identifying one or more regions that include at least one of (i) one or more areas of interest (AOI), (ii) one or more distractor regions, and (iii) one or more non-salient regions (NSR);generating segmented entities corresponding to the regions by utilizing one or more models that include a segmentation model, a classification model, or both, on the image data;determining visual or structural relationships between a combination of regions and segmented entities and scene contents excluding segmented entities; andgenerating or recommending scene modifications to one or more of the identified regions to achieve a user-defined visual attention goal.

2. The at least one non-transitory computer-readable medium instructions of claim 1 wherein determining visual or structural relationships between the segmented entities further includes linked regions exhibiting similarity.

3. The at least one non-transitory computer-readable medium instructions of claim 1 further including instructions for determining, for each segmented entity, one or more visual attention metrics based on saliency predictions or gaze sequence points produced by the VAM.

4. The at least one non-transitory computer-readable medium instructions of claim 1 further including instructions for performing an analysis to identify linked regions of the scene exhibiting visual or structural similarity to a segmented entity, using one or more feature detection methods or neural network-based matching.

5. The at least one non-transitory computer-readable medium instructions of claim 4 further including instructions for determining linked regions includes applying a similarity detection algorithm selected from Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), or convolutional neural network (CNN) feature matching to detect visually or structurally similar regions for synchronized editing.

6. The at least one non-transitory computer-readable medium instructions of claim 1 wherein executing the segmentation models is triggered based on a timer-based or event-based condition, including initiating the segmentation automatically after a predetermined time passes from a last user modification of an AOI, or upon detection of a user event in a graphical interface.

7. The at least one non-transitory computer-readable medium instructions of claim 1 wherein classifying segmented entities further includes applying multiple segmentation paradigms selected from instance segmentation, semantic segmentation, and panoptic segmentation, and merging their outputs to form a unified segmentation result.

8. The at least one non-transitory computer-readable medium instructions of claim 1 further including instructions for providing a user interface that displays segmented entities and allows the user to refine segmentation boundaries by supplying inclusion point prompts, exclusion point prompts, or mask inputs from external tools, and automatically rerunning segmentation when new inputs are received.

9. The at least one non-transitory computer-readable medium instructions of claim 1 further including instructions for generating scene modifications further includes constraining modifications to selected color or text change modes, including hue, brightness, saturation, font type, font size, and text orientation, according to user-defined value ranges or discrete allowed values.

10. The at least one non-transitory computer-readable medium instructions of claim 1 further including instructions for performing sensitivity analysis to determine the effect of incremental modifications in hue, brightness, or saturation on at least one attention score derived from the VAM, and using the sensitivity information to refine subsequent modification recommendations.

11. The at least one non-transitory computer-readable medium instructions of claim 1 wherein identification of distractor regions further includescalculating overlaps between predicted gaze sequence points or attention hotspots and segmented objects, anddetermining a region as a distractor when overlap area thresholds or distance criteria are satisfied.

12. The at least one non-transitory computer-readable medium instructions of claim 1 further including instructions for executing a scene optimization process that iteratively applies proposed modifications to AOIs, distractors, and NSRs, evaluates resultant saliency metrics from the VAM, and selects the modified scene that maximizes compliance with one or more user-defined visual attention goals.

13. At least one system includesat least one computing device that includes one or more processors; andat least one memory coupled to at least one of the one or more processors, wherein the at least one memory includes instructions that configure the at least one computing device toreceive image data representing a scene;predict regions associated with visual attention using a visual attention model (VAM) and identifying one or more regions that include at least one of (i) one or more areas of interest (AOI), (ii) one or more distractor regions, and (iii) one or more non-salient regions (NSR);generate segmented entities corresponding to the regions by utilizing one or more models that include a segmentation model, a classification model, or both, on the image data;determine visual or structural relationships between a combination of regions and segmented entities and scene contents excluding segmented entities; andgenerate or recommend scene modifications to one or more of the identified regions to achieve a user-defined visual attention goal.

14. The at least one system of claim 13 wherein determining visual or structural relationships between the segmented entities further includes linked regions exhibiting similarity.

15. The at least one system of claim 13 further including instructions for determining, for each segmented entity, one or more visual attention metrics based on saliency predictions or gaze sequence points produced by the VAM.

16. The at least one system of claim 13 further including instructions for performing an analysis to identify linked regions of the scene exhibiting visual or structural similarity to a segmented entity, using one or more feature detection methods or neural network-based matching.

17. The at least one system of claim 16 further including instructions for determining linked regions includes applying a similarity detection algorithm selected from Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), or convolutional neural network (CNN) feature matching to detect visually or structurally similar regions for synchronized editing.

18. The at least one system of claim 13 wherein executing the segmentation models is triggered based on a timer-based or event-based condition, including initiating the segmentation automatically after a predetermined time passes from a last user modification of an AOI, or upon detection of a user event in a graphical interface.

19. A computer-implemented method includesreceiving image data representing a scene;predicting regions associated with visual attention using a visual attention model (VAM) and identifying one or more regions that include at least one of (i) one or more areas of interest (AOI), (ii) one or more distractor regions, and (iii) one or more non-salient regions (NSR);generating segmented entities corresponding to the regions by utilizing one or more models that include a segmentation model, a classification model, or both, on the image data;determining visual or structural relationships between a combination of regions and segmented entities and scene contents excluding segmented entities; andgenerating or recommending scene modifications to one or more of the identified regions to achieve a user-defined visual attention goal.

20. The computer-implemented method of claim 19 wherein determining visual or structural relationships between the segmented entities further includes linked regions exhibiting similarity.

Images