Edge computing system for analyzing image data on satellite

Abstract

In some embodiments, a method receives image data of a location that was captured by a satellite based on a request. A prediction network generates a text embedding of the request and a set of image embeddings for the image data in a joint embedding space. The method selects a visual representation based on an analysis of the set of image embeddings and the text embedding in the joint embedding space. A text response is output for the visual representation based on an analysis of the visual representation and the request, wherein the text response is transmitted to a satellite command system.

Description

Attorney Docket No.: PLAB-109-PCTEDGE COMPUTING SYSTEM FOR ANALYZING IMAGE DATA ON SATELLITECROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The current application claims the benefit of U.S. Provisional Patent Application No. 63 / 637,730 filed on April 23, 2024, the disclosure of which is incorporated herein in its entirety.FIELD OF TECHNOLOGY

[0002] This patent document relates generally to image processing and more specifically to communicating responses from satellites to a ground-based command system based on the image processing.BACKGROUND

[0003] A constellation of imaging satellites can be used to acquire imagery. A request for imagery may be received. The satellites can be controlled to acquire the imagery by a ground-based command system. The command system can uplink commands to the satellites and receive imagery via a satellite downlink. The time to transmit the imagery via the downlink may experience long download times. The long download times may delay providing a response to the request. Also, the delay may affect the usefulness of imagery.BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The included drawings are for illustrative purposes and serve only to provide examples of possible structures and operations for the disclosed inventive systems, apparatus, methods and computer program products. These drawings in no way limit any changes in form and detail that may be made by one skilled in the art without departing from the spirit and scope of the disclosed implementations.

[0005] FIG. 1 depicts a simplified system for providing edge computing analysis of image data according to some embodiments.

[0006] FIG. 2 depicts an example of level 0 image data according to some embodiments.

[0007] FIG. 3 depicts a more detailed example of an Earth Observation (EO) satellite according to some embodiments.

[0008] FIG. 4 depicts a more detailed example of a prediction network according to some embodiments.|0009| FIG. 5 depicts a more detailed example of the prediction network according to some embodiments.10010] FIG. 6 depicts a more detailed example of image data that is input into prediction network according to some embodiments.

[0011] FIG. 7 depicts a simplified flowchart for processing image data according to some embodiments.

[0012] FIG. 8 depicts a simplified flowchart of a method for training the prediction network according to some embodiments.

[0013] FIG. 9 depicts an example of training an image encoder according to some embodiments.

[0014] FIG. 10 depicts an example of training a text encoder and image encoder together according to some embodiments.

[0015] FIG. 11 depicts examples of captions that could be used to the train text encoder and image encoder according to some embodiments.

[0016] FIG. 12 illustrates one example of a computing device according to some embodiments.[0017[ FIG. 13 depicts an example of an image model and a language model according to some embodiments.DETAILED DESCRIPTION[0018| Described herein are techniques for a satellite edge computing system. In the following description, for purposes of explanation, numerous examples and specific details are set forth to provide a thorough understanding of some embodiments. In some embodiments as defined by thePLAB-109-PCT 2claims may include some or all the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0019] A satellite may use an edge computing system to process image data that is acquired, such as by the satellite (or another satellite). In some embodiments, a request may be determined, such as a request may be received from a ground-based satellite command system. The request may be a text query, such as “Search for a number of airplanes in an airfield at a location #1.” The satellite may capture image data based on the location. In some embodiments, the image data may be unprocessed or raw data, but the image data may be processed with some limits. For example, the computing power on the satellite may not be used to fully process the image data compared to how the image data could be processed using a system on the ground that has more available computing resources. The image data and the text query may then be input into a prediction network. One or more image embeddings for the image data and a text embedding for the text query may be output by the prediction network in a joint embedding space. An image embedding may be selected as being associated with the text embedding for the text query. For example, for the text query above, the prediction network may predict portions of the image data that are showing airplanes based on an image embedding and the text embedding in the joint embedding space. A text response to the text query may then be output based on the image embedding that is selected. For example, the system may analyze the text query and the image embedding that is determined to be most similar to the text embedding in the joint embedding space to determine a caption that describes the content of the associated image data associated with the image embedding. In some embodiments, the caption for the image embedding may be “There are 15 airplanes parked on a runway surrounded by hangers and support buildings.”

[0020] The satellite may return the text response to the satellite command system via a downlink. In some embodiments, a small image chip of image data may also be provided. The image chip may be a portion of the image data that was collected, such as from image data for the visual representation that was determined to be relevant to the text embedding. If the image data that is analyzed is around 4 Gigabits, downloading the text response and the image chip may use less bandwidth than sending the entire set of image data. For example, on a 100 kilobit per second (kb / s) downlink, the download may take around three seconds to send the text response and imagePLAB-109-PCT 3chip compared to around 12 hours to download the entire set of image data that was collected. This results in an improvement in the latency in responding to a text query. Also, less resources may be used in sending the text response and image chip. Further, the reduced response time may make the information received more useful because it may be more timely received and can be acted upon sooner. This may be important in urgent situations that require urgent responses. Also, the prediction network is improved. The image data that is captured may need to be processed to increase the quality before analysis, such as before an object detection process could be run on the image data. For example, clouds may need to be removed among other processing to perform object detection. The satellite may have limited resources to process image data. To overcome this limitation, the prediction network may be trained to analyze unprocessed image data (or minimally processed data) to generate text responses. This saves computing resources by not requiring large amounts of processing on image data before analysis. For example, processing that is performed once the images have been downlinked to the ground may include image distortion correction, radiometric correction, stitching, mapping to ground control points, etc. Also, the text response may provide more information by providing a context of the image data that is received instead of just the image data.System

[0021] FIG. 1 depicts a simplified system 100 for providing edge computing analysis of image data according to some embodiments. System 100 includes a satellite command system 102, earth observation (EO) satellites 104, Low Earth Orbit (LEO) / Medium Earth Orbit (MEO) satellite 108, and a geostationary (GEO) satellite 106. Although a single LEO / MEO satellite 108 and a single GEO satellite 106 are shown, multiple instances of these satellites may be appreciated. Also, other structures of satellite communication systems may be appreciated.

[0022] EO satellites 104 may be imaging satellites that may form a constellation of imaging satellites. EO satellites 104 may include a memory for storing image data and communications equipment for communicating with satellite command system 102. The communication equipment may include steerable antenna arrays and / or reprogrammable radio communications equipment that can be configured to communicate with satellite command system 102 directly, through GEOPLAB-109-PCT 4satellite 106, or through LEO / MEO satellite 108. Although EO satellites 104 are described, different imaging satellites may be used.(0023] LEO / MEO satellite 108 may be located in low earth orbit (LEO) or medium earth orbit (MEO). In some embodiments, LEO / MEO satellite 108 may be located at a higher altitude than EO satellites 104. A constellation of LEO / MEO satellites 108 may be provided to provide wide coverage to enable selected EO satellites 104. LEO / MEO satellites 108 may be relay satellites that relay communications to / from EO satellite 104. In some examples, LEO / MEO satellites 108 may relay the communication to a ground-based station to provide a quicker downlink transmission (e.g., lower latency) than would be provided by waiting for EO satellite 104 to arrive in the appropriate position in its orbit to directly downlink to the ground-based station.

[0024] GEO satellite 106 may be a geostationary satellite that includes an orbit above Earth (or another body) that has a same duration as the Earth’s rotation period about its axis and therefore appears stationary above a ground station on Earth. Multiple GEO satellites 106 may be provided to generally provide line of sight coverage to the ground stations. EO satellites 104 are able to connect to a nearest GEO satellite to provide coverage over the entire Earth and provide access to all EO satellites 104 at all times. GEO satellite 106 and LEO / MEO satellite 108 may be associated with another company than the company that owns EO satellites 104, and communicate with an operator teleport station hub. The station hub may then communicate via a modem to satellite command system 102. The use of either GEO satellite 106 or LEO / MEO satellite 108 may provide near real-time communications from EO satellites 104 to satellite command system 102. In some embodiments, the communication via GEO satellite 106 or LEO / MEO satellite 108 may be faster than a communication from EO satellite 104 directly to satellite command system 102 without going through GEO satellite 106 or LEO / MEO satellite 108. These communication pathways via GEO satellite 106 or LEO / MEO satellite 108 can allow communications to and from EO satellite 104 in at least near real-time without any special pointing requirements (at any time), which may avoid delays in responding to requests. However, a communication may be transmitted directly to EO satellite 104 (e.g., without the use of an intermediate GEO / LEO / MEO satellite). In such a scenario, the pathway may require certain satellite pointing parameters to send or receive the communication to or from EO satellite 104 (e.g., when the satellite is within a certain range of a ground command center). Although the above paths are described, other paths may be appreciated.PLAB-109-PCT 5Additional details of paths may be found in U.S. Patent No. 10,979,137, entitled "Multi -Pathway Satellite Communication Systems and Methods”, filed August 1, 2019, and U.S. Patent No.11,463,158, entitled "Multi -Pathway Satellite Communication Systems and Methods”, filed September 29, 2020, all of which are incorporated by reference for all purposes.

[0025] Satellite command system 102 may be located in a geographic location, such as on Earth, another planet, a celestial body, a portion of the universe, etc. Satellite command system 102 may include a request processing system 112 that can receive a request for image data of a particular target, such as a geographic area on Earth, or another entity. The request may indicate a text query for a location of the target to be imaged. Satellite command system 102 may select an EO satellite 104 to process the request and send the request via an uplink to EO satellite 104. The request may be sent via different paths. For example, the request may be sent via GEO satellite 106 or LEO / MEO satellite 108. When sent via GEO satellite 106, the request is forwarded to GEO satellite 106, which then sends the request to EO satellite 104. Also, the request may be sent to LEO / MEO satellite 108, which forwards the request to EO satellite 104. Alternatively, the request may be sent directly to an EO satellite 104.

[0026] Previously, an EO satellite 104 may be selected to process the request. After collecting image data based on the target location, the collected image data was then downlinked to satellite command system 102. The downlink could be from EO satellite 104 directly to satellite command system 102, via GEO satellite 106 to satellite command system 102, or via LEO / MEO satellite 108 to satellite command system 102. In some examples, the amount of image data may be around 4 Gb or more, which may require a long amount of time to transfer via a downlink. For example, EO satellite 104 may be configured to pass over a ground station every X minutes, such as 90 minutes. EO satellite 104 may be able to use a higher speed downlink during this time to send the image data. GEO satellite 106 may be always accessible to EO satellite 104; however, the downlink speed from GEO satellite 106 may be slower, such as around 100 Kbps / second. In some examples, using a 100 Kbps / second downlink would take 12 hours to download around 4 Gb of image data. Transmitting the entire set of image data via GEO satellite 106 may not be feasible, and a link with GEO satellite 106 for 12 hours would not be maintained. A downlink between LEO / MEO satellite 108 and a ground station may be faster. However, EO satellite 104 may have to wait to connect with a LEO / MEO satellite 108. Accordingly, to improve the latency ofPLAB-109-PCT 6responding to the request, the following will describe using edge computing system 110 to send a text response to the request that describes relevant image data that is captured by EO satellite 104. The smaller response can leverage the use of alternative downlinks and send a response within a shorter time period than waiting for EO satellite 104 to pass over a ground station (or to connect with LEO / MEO satellite 108)

[0027] After receiving a request, EO satellite 104 may capture image data of a target location. Then, edge computing system 110 may receive the image data that is captured. In some embodiments, edge computing system 110 may process the image data on EO satellite 104. Also, edge computing system 110 may be located on another device in space, such as another satellite. However, for discussion purposes, edge computing system 110 is described as being located on the satellite that captured the image data.

[0028] To process the request, edge computing system 110 may use a prediction network to determine a text response to the text query based on the image data that is captured. As an example, if the text query is “How many airplanes are located at a target location?”, the prediction network may analyze the image data and text query, and determine a text response of “There are 15 airplanes parked on a runway surrounded by hangers and support buildings.”

[0029] The image data may be classified in different levels. For example, the levels may be rated from level 0 to level 10. Level 0 image data may be the raw data that is captured from a camera system. Each different level may be processed using different techniques. Some methods of processing may include cloud removal, band alignment, orthorectification, radiometric correction, and geolocation. The amount of processing that is required for each level may vary. Although these levels are described, the definition of levels may vary.

[0030] Traditionally, the image data is processed such that an analysis of the image data may be more accurate, such as object detection with clouds removed is more accurate. Satellite command system 102 may perform the processing of the image data on the ground. This leverages more available computing resources on the ground compared to the computing resources located on EO satellites 104. Not only are EO satellites 104 space constrained, but computer processors may be processing other tasks for the satellite. The processing on the ground may still be performed and is needed for insights other than the response to the text query. However, the processing on thePLAB-109-PCT 7ground cannot be completed in the time that is taken to generate the text response (or even the image data transmitted to the ground in that time).(0031] In some embodiments, the full ground-based processing described above may not be possible on EO satellite 104 due to limited computing resources (e.g., EO satellite 104 may not have enough space to include the computing resources required). EO satellite 104 may perform the edge processing on image data that has not been processed (raw image data) or that has been minimally processed on EO satellite 104. The ability of edge computing system 110 to analyze the image data that has been not processed or minimally processed may allow text responses to text queries to be generated without the full ground-based processing. This may be an improvement to overcome the restriction of limited computing resources that are available on EO satellite 104. For example, the prediction network may be trained to answer text queries that have clouds that have not been removed (e.g., cloud removal may take several passes and cloud-free days).

[0032] After edge computing system 110 determines the text response, the text response can be returned to satellite command system 102 much faster than the image data that was analyzed. For example, a small image chip of the image data (e.g., a portion of an image) and the text response may be returned on the 100 kb / s downlink in around three seconds via GEO satellite 106 or LEO / MEO satellite 108. EO satellite 104 may also return the response much quicker than the image data directly to satellite command system 102. The generation of the text response will be described in more detail below.

[0033] Request processing system 112 may then process the response. For example, request processing system 112 may communicate with the requester to provide the text response and the image chip. The response time may be much improved compared to receiving the image data from EO satellite 104, processing the image data into higher level data, and then performing an object detection process on the images to determine the answer to the query.

[0034] The following will now describe the processing of the image data in more detail.Image Data Processing

[0035] FIG. 2 depicts an example of level 0 image data according to some embodiments. The level 0 image data may be collected in a minimum unit of collect (MUC). Each minimum unit of connectPLAB-109-PCT 8may be shown in one of strips 202-1 to 202-8. Strips 202-1 to 202-8 may be considered level 0 data. Previously, strips 202-1 to 202-8 may be processed to stitch the strips together to form an image. Other processing may be performed such as alignment of the strips, orthorectification, geolocation, cloud removal, and radiometric correction. Then, the resulting image may be analyzed, such as by performing object detection. However, edge computing system 110 may be configured to process level 0 image data directly. Although processing level 0 image data is described, some image processing may be performed on EO satellite 104. That processing on EO satellite 104 may be less than the processing that may be performed to form an image on the ground previously.EO satellite

[0036] FIG. 3 depicts a more detailed example of EO satellite 104 according to some embodiments. Camera 302 may be any device that can capture images of the target location from the request. The captured image data may be considered level 0 data and is provided to edge computing system 110. As discussed above, some processing of the captured image data may be performed. However, for discussion purposes, the level 0 data is processed. The captured image data may be input into a prediction network 304 with the text query that is received from the request. The text query may be received via a real-time link, such as via GEO satellite 106. Prediction network 304 may analyze the text query and the captured image data to determine a response. As will be described in more detail below, prediction network 304 includes multiple models that perform different functions. Prediction network 304 may determine a text embedding from the text query and image embeddings from the captured image data in a joint embedding space. Prediction network 304 then generates a response using the image embeddings and the text embedding. The response may be a caption that describes content of corresponding image data that is relevant to the text embedding, and may optionally include an image chip of a portion of image data that was captured. The response is provided to a communication system 306, which may send the response to satellite command system 102 in near real-time. As discussed above, the response may be sent via multiple different paths.Prediction NetworkPLAB-109-PCT 9

[0037] FIG. 4 depicts a more detailed example of prediction network 304 according to some embodiments. Although this structure of prediction network 304 is described, other structures may be appreciated. Prediction network 304 includes a text encoder 402 and an image encoder 404.10038] Text encoder 402 receives a text query and can output a text embedding. The text embedding may be a representation of the text query. For example, text encoder 402 may extract important features of the text query to generate the text embedding. Different text encoders may be used.

[0039] Image encoder 404 may generate image embeddings for respective captured image data. Image encoder 404 extracts the important features from image data into a representation of the features, which is referred to as an image embedding. As will be described below, portions of the image data referred to as image chips may be sequentially input into image encoder 404, which outputs a sequence of image embeddings.

[0040] The text embedding and image embeddings may be located in a joint embedding space 406. Joint embedding space 406 may connect text embeddings and image embeddings in the space. Also, joint embedding space 406 may be a higher dimensional space with the dimensionality of around 256 to 1024 dimensions. In general, joint embedding space 406 may associate visual concepts in images with text. This creates a joint embedding space that correlates text embeddings and image embeddings.

[0041] An image model 408 may identify the most relevant visual information based on the text query from joint embedding space 406. In some embodiments, image model 408 may compute a similarity of image embeddings to the text embedding in joint embedding space 406 to select an image embedding. For example, a similarity calculation, such as a cosine similarity or nearest neighbor search, is computed between image embeddings and the text embedding to select an image embedding that is closest in distance to the text embedding in joint embedding space 406. In some embodiments, image model 408 may use a cross-attention module. Cross-attention allows image model 408 to identify the most relevant visual information based on the text query. The cross-attention mechanism may be used as follows:Q cross-W qcross • text_emb eddi ngPLAB-109-PCT 10K c ro ss W kero s s ' image_embeddingsV cross-Wvcross ' image_embeddingsattended_visual=atention_scorescross Vcross

[0042] Here, Qcross represents the query (text embedding), Kcross represents the keys (image embeddings), and Vcross represents the values (image embeddings). The attention scores are computed by taking the dot product of Qcross and Kcross, followed by a softmax activation. The attended visual representation is obtained by multiplying the attention scores with Kcross.

[0043] The attended visual representation, which now encodes the relevant visual information based on the query is then fed into a language model along with the text embedding. A language model 410 may then output a text response for the image embedding. Different processes may be used to generate the text response. For example, language model 410 may be trained to generate text that describes the associated image data for the selected image embedding. As described above, if the image embedding is associated with a picture of airplanes and the text embedding is asking how many airplanes are shown, the text response for the image data may be “There are 15 airplanes shown.” Importantly, this response may be generated without running an object detection process on the image data. In other embodiments, the image data for the image embedding along with the text query may be input into another prediction network that is trained to analyze the content of the image data. The output of the prediction network may describe the content of the image data based on the text query.

[0044] In some embodiments, the attended visual representation is first projected into the dimensionality of the text embedding using a fully connected (linear) layer. It is then added element-wise to the text embedding. The language model employs self-attention to capture the dependencies within the textual input (combined with the attended visual input) and generate a coherent response. The attended visual combined with text also is called the text representation. The self-attention mechanism can be formulated as follows:PLAB-109-PCT 11Qseif=W qseif • text_embeddi ngKseif=W kseif • text_emb eddingVseif=Wvseif- text_embeddingattended_text=attention_scores_self Vseif

[0045] The attention mechanism is similar to that used by image model 408, except here is using self-attention instead of cross-attention, and hence the same inputs (text embedding) to Q, K, and V are used. The self-attended text representation is then passed through fully connected (linear layers) to generate the output text response “I see two Su-27 fighter parked on the run-way,” accurately describing the visual content of the satellite images based on the input query of “Describe the airfield at location X” . FIG. 13 depicts an example of image model 408 and language model 410 according to some embodiments.0046] By combining visual and text embeddings through cross-attention and utilizing a language model with self-attention, the system can effectively combine text queries with satellite imagery and generate relevant textual responses. This approach leverages the strengths of both visual and linguistic representations to provide accurate and informative responses to user queries in the context of analyzing satellite images over different content. The network is pre-trained using paired image-text pairs and fine-tuned on a smaller set of responses to questions on satellite imagery.

[0047] FIG. 5 depicts a more detailed example of prediction network 304 according to some embodiments. Although the following inputs to prediction network 304 are described, other inputs may be used. At 502, the image data is received. For example, the image data may be panchromatic (PAN) image data, which may be grayscale or black and white data that is captured via the camera. The PAN image data may be collected in minimum units of collect that may form an image of a certain size (e.g., 12k x 12k pixels). At 504, the image data may be broken into smaller chips that may be input into image encoder 404. For example, the 12k x 12k pixel image may be broken intoPLAB-109-PCT 12200x200 pixel image chips may be sequentially input into image encoder 404 to create image embeddings at 506.(0048] At 508, a text query is received that indicates a request of “Search for airplanes at location #1.” The text query may be received from satellite command system 102 in near real-time. The text query may also be preset on EO satellite 104. For example, without receiving a new request, EO satellite 104 may monitor captured image data for a number of airplanes. At 510, text encoder 402 may generate a text embedding based on the text query.

[0049] Edge computing system 110 determines visual information that is considered similar to the text embedding. Different computations may be used. Then, a caption associated with the visual information and the text embedding may be generated, which may be “There are 15 airplanes found at location #1.”

[0050] FIG. 6 depicts a more detailed example of image data that is input into prediction network 304 according to some embodiments. At 602, a strip collect from EO satellite 104 includes multiple minimum units of collect over time. In some embodiments, a minimum unit of collect 602-1 is shown in an expanded view for discussion purposes. Additional minimum units of collect are captured and shown thereafter.[00511 In some embodiments, a minimum unit of collect may include two PAN bands of around 12k x 12k pixels and six multispectral bands of around 12k x 12k pixels. Although this composition of image data is described, different formats of image data may be captured. Panchromatic data may capture light across the entire visible spectrum without discriminating between wavelengths. Multispectral bands may capture data across multiple spectral bands, which may include red, green, blue, etc. bands. The PAN bands and multispectral bands may be formed into fdes of a certain size, such as 4k x 3k pixel fdes. The fdes may be PAN Spectral Field Files (SFF), and there may be 12 spectral field fdes per minimum unit of collect.

[0052] At 610, a single spectral field file may then be broken up into smaller image chips. For example, the image chips each may be 200 x 200 pixels, and there is a total of 20*15*12=3600 image chips per minimum unit of collect. These image chips may each be input into prediction network 304, which outputs a corresponding image embedding for each image chip.PLAB-109-PCT 13

[0053] In some embodiments, a strip collect may include three minimum units of collection and may require two seconds to capture and covers a ground track of around 15 km. The strip collect may include 4.35 gigabits of image data. If the forward link is 20 bits per second unpointed (e.g., via LEO / MEO satellite 108) or 250 bits per second pointed (e.g., via GEO satellite 106), and the reverse link is 100 kb / s, the strip collect may take around 12 hours to capture and download on a lOOkb / s downlink. However, the output of a text caption and small 200x200 image chip may take around 3 seconds to download on a 100 kb / s downlink.

[0054] The following describes an overall process of processing image data and a text query. FIG.7 depicts a simplified flowchart 700 for processing image data according to some embodiments. At 702, edge computing system 110 determines a text query. The text query may be sent in a request from satellite command system 102. In other embodiments, the text query may be determined automatically. For example, a text query may be generated to look for changes in image data over a period of time. However, the automated text query may be generated based on a request from satellite command system 102 at a prior time.|0055| At 704, edge computing system 110 receives captured image data from EO satellite 104. As described above, the image data may be level 0 image data that is captured for a location associated with the text query. Also, the image data may be processed before analysis.

[0056] At 706, edge computing system 110 determines portions of the image data. For example, edge computing system 110 generates multiple spectral field files from the image data. From the spectral field files, edge computing system 110 may generate multiple image chips.|0057] At 708, edge computing system 110 inputs the text query and the portions of image data into prediction network 304 to generate a text embedding and image embeddings in joint embedding space 406. For example, edge computing system 110 may input the text query into text encoder 402 and the image chips into image encoder 404. This generates a text embedding and multiple image embeddings.

[0058] At 710, edge computing system 110 may analyze the image embeddings and text embedding to determine a visual representation. At 712, edge computing system 110 may analyzePLAB-109-PCT 14the visual representation and the text embedding to determine a text representation. At 714, a text response is generated for the text representation.(0059] The text response may be a caption describing information in associated image data from the visual information that is considered most relevant to the text embedding. For example, the caption may describe content in the image data that answers the text query. Then, at 716, edge computing system 110 outputs the text response and optionally a small image chip for the caption for near real-time communications. EO satellite 104 may then send the response to satellite command system 102. As discussed above, the response may be sent through multiple pathways. Whichever pathway is used, because the text of the caption and the small image chip is less than the image data that was analyzed, the response time may be drastically faster compared to sending the image data to satellite command system 102.

[0060] Prediction network 304 may be specially trained to run on EO satellite 104. As will be described, EO satellite 104 may have limited computing resources compared to satellite command system 102 and optimally uses the limited computing resources. Accordingly, prediction network 304 may be trained to process image data that has not been processed, such as level 0 data, or trained to process minimally processed data using the limited computing resources. The following describes the training process in more detail.Training(00611 FIG. 8 depicts a simplified flowchart 800 of a method for training prediction network 304 according to some embodiments. Although this method of training is described, other methods may be appreciated. At 802, in a first training stage, image encoder 404 is trained separately from text encoder 402. Image encoder 404 may be trained using different processes. In some embodiments, the parameters of image encoder 404 may be pre-trained and the values of the parameters are refined, but image encoder 404 does not need to be pre-trained. In the training process, image encoder 404 may be trained using masked images that are reconstructed by image encoder 404 and a decoder. The parameters of image encoder 404 may be adjusted based on the reconstruction. This process will be described in more detail in FIG. 9. Other methods for training image encoder 404 may also be appreciated.PLAB-109-PCT 15

[0062] At 804, after training image encoder 404, image encoder 404 is attached to text encoder 402 for a second training stage. Parameters for both text encoder 402 and image encoder 404 may be trained in the second training stage. In the training, at 806, text captions for images are created. The text captions may be automatically generated based on analysis of content in the images or may be user generated. Examples of captions may describe the content found in the images. Some examples of captions are shown in FIG. 11.

[0063] At 808, the captions are input into text encoder 402 and respective images are input into image encoder 404. The output may be a caption for respective images. For example, if an image of four airplanes in an airfield are shown, the caption may be four white planes are between white buildings. The process will be described in more detail in FIG. 10.

[0064] At 810, text encoder 402 and image encoder 404 are trained based on the caption that was output. For example, depending on the similarity of the caption that is output and the content of the image, parameters of text encoder 402 and image encoder 404 may be adjusted. The adjustment of parameters may be based on contrastive loss, which will be described below in FIG. 10.

[0065] FIG. 9 depicts an example of training image encoder 404 according to some embodiments. The following may be used to train image encoder 404 to process level 0 data. For example, level 0 data may have clouds to occlude objects, or other imperfections that affect an analysis. By training image encoder 404 on masked content that may be similar to level 0 data, image encoder 404 may be able to process level 0 data more accurately. At 904, an image with masking is received. The image may be higher level image data than the image data that will be analyzed by image encoder 404 on EO satellite 104. The masking may mask portions of the image, such as at least 75% of the image although other percentages may be used. The masked portions may be missing or occluded regions. The masking may occlude or alter the content of the image, such as setting pixel values to a constant value or removing content. Image encoder 404 may generate an image embedding as described above based on the masked image. A decoder 902 is attached to image encoder 404 in the training stage, but may not be used after the training stage to process text queries. Decoder 902 is configured to receive an image embedding from generated by image encoder 404, and generate a reconstructed image. For example, decoder 902 may output the imagePLAB-109-PCT 16shown at 906, which includes content where the masked portions existed in the image inputted into image encoder 404.(0066] In the above training, image encoder 404 encodes the masked image into a lower dimensional latent representation of an image embedding. The image embedding captures the important features of the visible parts of the masked image and ignores the missing or occluded regions. Decoder 902 receives the image embedding and decodes it to produce the reconstructed image. Decoder 902 may generate predictions for the missing or occluded regions of the image based on the image embedding. The training of image encoder 404 may adjust parameters to minimize the discrepancy between the reconstructed image and the unmasked image. Various loss functions may be used to determine the difference. The parameters of image encoder 404 (and decoder 902) may be optimized to minimize the loss between the reconstructed image and the unmasked image. This may train image encoder 404 to generate image embeddings using image data such as level 0 data, which may include occluded areas. Once image encoder 404 is trained, in operation, decoder 902 may not be used. Rather, image encoder 404 receives image data that is unmasked and can generate image embeddings as described above. Also, parameters of image encoder 404 may be further refined using level 0 images, such as by masking the level 0 images.

[0067] For the second training stage, FIG. 10 depicts an example of training text encoder 402 and image encoder 404 together according to some embodiments. At 1002, a sequence of image, text pairs {Ii, Ti, I2, T2, ... , IN, TN} is input into text encoder 402 and image encoder 404, respectively. The caption may be text information that describes content found in the image, or is the desired text to be associated with the image. For example, the caption may be “A shipping container terminal with 2 docked ships” if the content shows two ships docked at a terminal.(0068] Text encoder 402 may output a text embedding for the caption and image encoder 404 outputs an image embedding for the image. A training process may be used, such as contrastive loss, which may adjust the parameters of text encoder 402 and image encoder 404 to encourage the text embedding and the image embedding to be closer together in joint embedding space 406 in positive pairs (e.g., the caption correctly describes content in the image) and push the text embedding and image embedding of negative pairs (e.g., the caption incorrectly describes content in the image) further apart in joint embedding space 406. Eventually, text encoder 402 and imagePLAB-109-PCT 17encoder 404 are trained to output a text embedding and an image embedding that are close together in joint embedding space 406 when positive pairs are encountered and farther away when negative pairs are encountered.

[0069] At 1006, multiple text embeddings are Ti, T2, T3, TN. At 1008, multiple image embeddings for multiple images for the respective text embeddings are Ii, I2, I3, ..., IN. The image embeddings and text embeddings with the same number may be positive pairs and image embeddings and text embeddings with different numbers may be negative pairs. Text encoder 402 and image encoder 404 are trained to associate respective text embeddings and image embeddings to be closer in joint embedding space 406 for positive pairs. That is, image embedding Ii is associated to be closer to text embedding Ti in joint embedding space 406, image embedding I2 is closer together with text embedding T2 in joint embedding space 406, and so on. These embeddings are closer in joint embedding space 406 compared to other image embedding and text embedding combinations (e.g., T2 and I3).

[0070] FIG. 11 depicts examples of captions that could be used to train text encoder 402 and image encoder 404 according to some embodiments. At 1102, an image is shown of an airfield. At 1104, captions for the content shown in the image are provided, such as "A lineup of Sukhoi 24 bombers with military camouflage parked on the concrete apron1', "A network of taxiways crisscrossing an airfield connecting hangars and runways with Sukhoi 24s parked alongside", and “Six Sukhoi 24 aircraft parked on the runway, with hardened aircraft shelters and maintenance areas”. Although these images shown are processed images, level 0 image data may be used with the associated captions. Using the level 0 image data may train prediction network 304 to associate captions with level 0 image data.

[0071] In addition to the above training, other training may be performed to refine the parameters of text encoder 402 or image encoder 404. For example, image encoder 404 may be used to identify zero-shot concepts that were not present in the training data. In zero-shot training, a text query is input into text encoder 402, such as “Show an image of a UFO”. Text encoder 402 may generate a text embedding from the text query. Prediction network 304 may output an image for the text embedding based on an image embedding that is considered to be closest to the text embedding in joint embedding space 406. For example, prediction network 304 may output an image with a red-PLAB-109-PCT 18green-blue smudge that indicates a fast moving spherical object that could conceivably be a UFO. Other training may also be performed by inputting an image into image encoder 404 and outputting text for the image. The text may then be analyzed to determine and adjust parameters of image encoder 404.

[0072] Accordingly, with the above techniques, prediction network 304 may be trained to analyze level 0 image data efficiently. This improves prediction network 304 and allows the processing of image data on EO satellite 104.Conclusion

[0073] The latency of delivering a response from EO satellites 104 to satellite command system 102 may be reduced significantly using edge computing system 110. Also, the amount of resources that are used to transmit the response is reduced. The context of the response may additionally be improved using the text description of the image data that is captured.System

[0074] FIG. 12 illustrates one example of a computing device according to some embodiments. According to various embodiments, a system 1200 suitable for implementing embodiments described herein includes a processor 1201, a memory 1203, a storage device 1205, an interface 1211, and a bus 1215 (e.g., a PCI bus or other interconnection fabric.) System 1200 may operate as a variety of devices such as edge computing system 110, or any other device or service described herein. Although a particular configuration is described, a variety of alternative configurations are possible. The processor 1201 may perform operations such as those described herein. Instructions for performing such operations may be embodied in the memory 1203, on one or more non-transitory computer readable media, or on some other storage device. Various specially configured devices can also be used in place of or in addition to the processor 1201. Memory 1203 may be random access memory (RAM) or other dynamic storage devices. Storage device 1205 may include a non-transitory computer-readable storage medium holding information, instructions, or some combination thereof, for example instructions that when executed by the processor 1201, cause processor 1201 to be configured or operable to perform one or more operations of a method as described herein. Bus 1215 or other communication components may support communicationPLAB-109-PCT 19of information within system 1200. The interface 1211 may be connected to bus 1215 and be configured to send and receive data packets over a network. Examples of supported interfaces include, but are not limited to: Ethernet, fast Ethernet, Gigabit Ethernet, frame relay, cable, digital subscriber line (DSL), token ring, Asynchronous Transfer Mode (ATM), High-Speed Serial Interface (HSSI), and Fiber Distributed Data Interface (FDDI). These interfaces may include ports appropriate for communication with the appropriate media. They may also include an independent processor and / or volatile RAM. A computer system or computing device may include or communicate with a monitor, printer, or other suitable display for providing any of the results mentioned herein to a user.Example Embodiments

[0075] In a 1st Example, a method comprising: receiving image data of a location that was captured by a satellite based on a request; generating, using a prediction network, a text embedding of the request and a set of image embeddings for the image data in a joint embedding space; selecting a visual representation based on an analysis of the set of image embeddings and the text embedding in the joint embedding space; and outputting a text response for the visual representation based on an analysis of the visual representation and the request, wherein the text response is transmitted to a satellite command system.

[0076] In a 2nd Example, the method of Example 1, wherein: the satellite command system is located on a ground of a celestial body and the satellite is orbiting the celestial body, the satellite command system uses an uplink to send the request to the satellite, and the satellite uses a downlink to send the text response to the satellite command system.

[0077] In a 3rd Example, the method of Example 2, wherein: uplink or downlink is via another satellite to the satellite.

[0078] In a 4th Example, the method of any of Examples 1-3, wherein: the image data comprises strips of data of the location, the strips of data are combined in a file, and image chips from the file are input into the prediction network to generate image embeddings for the image chips.

[0079] In a 5th Example, the method of any of Examples 1-4, wherein the image data is unprocessed image data from a camera.PLAB-109-PCT 20

[0080] In a 6th Example, the method of any of Examples 1-5, wherein: the text embedding is generated using a text encoder that is trained to generate text embeddings in the joint embedding space, and the set of image embeddings is generated using an image encoder that is trained to generate image embeddings in the joint embedding space.

[0081] In a 7th Example, the method of any of Examples 1-6, wherein generating the text response comprises: determining a text representation based on the visual representation and the text embedding; and generating the text response based on the text representation.

[0082] In an 8th Example, the method of any of Examples 1-7, wherein selecting the visual representation comprises: comparing the set of image embeddings and the text embedding in the joint embedding space; and selecting the visual representation that is considered most relevant to the text embedding.

[0083] In a 9th Example, the method of any of Examples 1-8, further comprising: during training of the prediction network, performing: generating image embeddings of masked images using an image encoder; generating reconstructed images from the image embeddings; and training parameters of the image encoder based on the reconstructed images and respective unmasked images.

[0084] In a 10th Example, the method of any of Examples 1-9, further comprising: during training of the prediction network, performing: inputting images into an image encoder to generate image embeddings; inputting captions that correspond to the respective images into a text encoder to generate text embeddings; and training parameters of the image encoder and the text encoder based on the respective image embeddings that correspond to text embeddings.

[0085] In a 11th Example, the method of Example 10, wherein the training parameters of the image encoder and the text encoder are trained to generate an image embedding and a text embedding that are farther away in the joint embedding space when the image and caption are negative pairs, and the training parameters of the image encoder and the text encoder are trained to generate an image embedding and a text embedding that are closer in the joint embedding space when the image and caption are positive pairs.PLAB-109-PCT 21

[0086] In a 12th Example, the method of any of Examples 1-11, further comprising: generating the text response based on a combination of the request and the visual representation.

[0087] In a 13th Example, the method of any of Examples 1-12, further comprising: sending the text response to an intermediary satellite for transmission to the satellite command system.|0088| In a 14th Example, the method of any of Examples 1-13, further comprising: determining a portion of the image data that is associated with the visual representation; and sending the text response and the portion of the image data to the satellite command system.

[0089] In a 15th Example, the method of any of Examples 1-14, wherein the joint embedding space associates visual concepts with captions during a training process.

[0090] In a 16th Example, a non-transitory computer-readable storage medium having stored thereon computer executable instructions, which when executed by a computing device, cause the computing device to be operable for: receiving image data of a location that was captured by a satellite based on a request; generating, using a prediction network, a text embedding of the request and a set of image embeddings for the image data in a joint embedding space; selecting a visual representation based on an analysis of the set of image embeddings and the text embedding in the joint embedding space; and outputting a text response for the visual representation based on an analysis of the visual representation and the request, wherein the text response is transmitted to a satellite command system.

[0091] In a 17th Example, the non-transitory computer-readable storage medium of Example 16, wherein: the satellite command system is located on a ground of a celestial body and the satellite is orbiting the celestial body, the satellite command system uses an uplink to send the request to the satellite, and the satellite uses a downlink to send the text response to the satellite command system.|0092| In an 18th Example, the non-transitory computer-readable storage medium of any of Examples 16-17, further operable for: during training of the prediction network, performing: generating image embeddings of masked images using an image encoder; generating reconstructed images from the image embeddings; and training parameters of the image encoder based on the reconstructed images and respective unmasked images.PLAB-109-PCT 22

[0093] In a 19th Example, the non-transitory computer-readable storage medium of any of Examples 16-18, further operable for: determining a portion of the image data that is associated with the image embeddings; and sending the text response and the portion of the image data to the satellite command system.

[0094] In a 20th Example, an apparatus comprising: one or more computer processors; and a computer-readable storage medium comprising instructions for controlling the one or more computer processors to be operable for: receiving image data of a location that was captured by a satellite based on a request; generating, using a prediction network, a text embedding of the request and a set of image embeddings for the image data in a joint embedding space; selecting a visual representation based on an analysis of the set of image embeddings and the text embedding in the joint embedding space; and outputting a text response for the visual representation based on an analysis of the visual representation and the request, wherein the text response is transmitted to a satellite command system.

[0095] Any of the disclosed implementations may be embodied in various types of hardware, software, firmware, computer readable media, and combinations thereof. For example, some techniques disclosed herein may be implemented, at least in part, by non-transitory computer-readable media that include program instructions, state information, etc., for configuring a computing system to perform various services and operations described herein. Examples of program instructions include both machine code, such as produced by a compiler, and higher-level code that may be executed via an interpreter. Instructions may be embodied in any suitable language such as, for example, Java, Python, C++, C, HTML, any other markup language, JavaScript, ActiveX, VBScript, or Perl. Examples of non-transitory computer-readable media include, but are not limited to: magnetic media such as hard disks and magnetic tape; optical media such as flash memory, compact disk (CD) or digital versatile disk (DVD); magneto-optical media; and other hardware devices such as read-only memory (“ROM”) devices and random-access memory (“RAM”) devices. A non-transitory computer-readable medium may be any combination of such storage devices.

[0096] In the foregoing specification, various techniques and mechanisms may have been described in singular form for clarity. However, it should be noted that some embodiments includePLAB-109-PCT 23multiple iterations of a technique or multiple instantiations of a mechanism unless otherwise noted. For example, a system uses a processor in a variety of contexts but can use multiple processors while remaining within the scope of the present disclosure unless otherwise noted. Similarly, various techniques and mechanisms may have been described as including a connection between two entities. However, a connection does not necessarily mean a direct, unimpeded connection, as a variety of other entities (e.g., bridges, controllers, gateways, etc.) may reside between the two entities.

[0097] Some embodiments may be implemented in a non-transitory computer-readable storage medium for use by or in connection with the instruction execution system, apparatus, system, or machine. The computer-readable storage medium contains instructions for controlling a computer system to perform a method described by some embodiments. The computer system may include one or more computing devices. The instructions, when executed by one or more computer processors, may be configured or operable to perform that which is described in some embodiments.

[0098] As used in the description herein and throughout the claims that follow, “a”, “an”, and “the” includes plural references unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise.

[0099] The above description illustrates various embodiments along with examples of how aspects of some embodiments may be implemented. The above examples and embodiments should not be deemed to be the only embodiments and are presented to illustrate the flexibility and advantages of some embodiments as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations, and equivalents may be employed without departing from the scope hereof as defined by the claims.PLAB-109-PCT 24

Claims

CLAIMSWhat is claimed is:

1. A method comprising:receiving image data of a location that was captured by a satellite based on a request;generating, using a prediction network, a text embedding of the request and a set of image embeddings for the image data in a joint embedding space;selecting a visual representation based on an analysis of the set of image embeddings and the text embedding in the joint embedding space; andoutputting a text response for the visual representation based on an analysis of the visual representation and the request, wherein the text response is transmitted to a satellite command system.

2. The method of claim Error! Reference source not found., wherein: the satellite command system is located on a ground of a celestial body and the satellite is orbiting the celestial body,the satellite command system uses an uplink to send the request to the satellite, andthe satellite uses a downlink to send the text response to the satellite command system.

3. The method of claim Error! Reference source not found., wherein: uplink or downlink is via another satellite to the satellite.

4. The method of claim Error! Reference source not found., wherein: the image data comprises strips of data of the location,the strips of data are combined in a file, andPLAB-109-PCT 25image chips from the file are input into the prediction network to generate image embeddings for the image chips.

5. The method of claim Error! Reference source not found., wherein the image data is unprocessed image data from a camera.

6. The method of claim Error! Reference source not found., wherein: the text embedding is generated using a text encoder that is trained to generate text embeddings in the joint embedding space, andthe set of image embeddings is generated using an image encoder that is trained to generate image embeddings in the joint embedding space.

7. The method of claim Error! Reference source not found., wherein generating the text response comprises:determining a text representation based on the visual representation and the text embedding; andgenerating the text response based on the text representation.

8. The method of claim Error! Reference source not found., wherein selecting the visual representation comprises:comparing the set of image embeddings and the text embedding in the joint embedding space; andselecting the visual representation that is considered most relevant to the text embedding.

9. The method of claim Error! Reference source not found., further comprising:during training of the prediction network, performing:PLAB-109-PCT 26generating image embeddings of masked images using an image encoder;generating reconstructed images from the image embeddings; and training parameters of the image encoder based on the reconstructed images and respective unmasked images.

10. The method of claim Error! Reference source not found., further comprising:during training of the prediction network, performing:inputting images into an image encoder to generate image embeddings; inputting captions that correspond to the respective images into a text encoder to generate text embeddings; andtraining parameters of the image encoder and the text encoder based on the respective image embeddings that correspond to text embeddings.

11. The method of claim Error! Reference source not found., wherein the training parameters of the image encoder and the text encoder are trained to generate an image embedding and a text embedding that are farther away in the joint embedding space when the image and caption are negative pairs, andthe training parameters of the image encoder and the text encoder are trained to generate an image embedding and a text embedding that are closer in the joint embedding space when the image and caption are positive pairs.

12. The method of claim Error! Reference source not found., further comprising:generating the text response based on a combination of the request and the visual representation.

13. The method of claim Error! Reference source not found., further comprising:PLAB-109-PCT 27sending the text response to an intermediary satellite for transmission to the satellite command system.

14. The method of claim Error! Reference source not found., further comprising:determining a portion of the image data that is associated with the visual representation; andsending the text response and the portion of the image data to the satellite command system.

15. The method of claim Error! Reference source not found., wherein the joint embedding space associates visual concepts with captions during a training process.

16. A non-transitory computer-readable storage medium having stored thereon computer executable instructions, which when executed by a computing device, cause the computing device to be operable for:receiving image data of a location that was captured by a satellite based on a request;generating, using a prediction network, a text embedding of the request and a set of image embeddings for the image data in a joint embedding space;selecting a visual representation based on an analysis of the set of image embeddings and the text embedding in the joint embedding space; andoutputting a text response for the visual representation based on an analysis of the visual representation and the request, wherein the text response is transmitted to a satellite command system.

17. The non-transitory computer-readable storage medium of claim Error! Reference source not found., wherein:PLAB-109-PCT 28the satellite command system is located on a ground of a celestial body and the satellite is orbiting the celestial body,the satellite command system uses an uplink to send the request to the satellite, andthe satellite uses a downlink to send the text response to the satellite command system.

18. The non-transitory computer-readable storage medium of claim Error! Reference source not found., further operable for:during training of the prediction network, performing:generating image embeddings of masked images using an image encoder; generating reconstructed images from the image embeddings; and training parameters of the image encoder based on the reconstructed images and respective unmasked images.

19. The non-transitory computer-readable storage medium of claim Error! Reference source not found., further operable for:determining a portion of the image data that is associated with the image embeddings; andsending the text response and the portion of the image data to the satellite command system.

20. An apparatus comprising:one or more computer processors; anda computer-readable storage medium comprising instructions for controlling the one or more computer processors to be operable for:receiving image data of a location that was captured by a satellite based on a request;generating, using a prediction network, a text embedding of the request and a set of image embeddings for the image data in a joint embedding space;PLAB-109-PCT 29selecting a visual representation based on an analysis of the set of image embeddings and the text embedding in the joint embedding space; andoutputting a text response for the visual representation based on an analysis of the visual representation and the request, wherein the text response is transmitted to a satellite command system.PLAB-109-PCT 30

Images