Systems and methods for phenomenon detection

Abstract

Systems and methods for phenomenon detection are presented. For example, a computing system may be configured to obtain image data from one or more surveillance image sensors of a satellite. The computing system can identify, based on the image data, a phenomenon within at least a portion of a field of view of the one or more surveillance image sensors. The computing system can generate, based on identifying the phenomenon within the at least the portion of the field of view of the one or more surveillance image sensors, a command to capture an image with a payload image sensor having a field of view smaller than the field of view of the one or more surveillance image sensors.

Description

SYSTEMS AND METHODS FOR PHENOMENON DETECTIONPRIORITY CLAIM

[0001] The present application is based on and claims priority to United States Provisional Application 63 / 748,610 having a filing date of January 23, 2025, which is incorporated by reference herein.FIELD

[0002] The present disclosure relates generally to techniques to detect phenomena, such as on slew trajectories of satellites. More particularly, the present disclosure relates to systems and methods for detecting phenomena using one or more surveillance image sensors to cause a payload image sensor to capture an image of the phenomena.BACKGROUND

[0003] A constellation of imaging satellites can be utilized to acquire imagery. The satellites can be controlled to acquire the imagery by, for example, a ground-based control center. The control center can uplink commands to the satellites and receive imagery via a satellite downlink.SUMMARY

[0004] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0005] One example embodiment of the present disclosure is directed to a computer system comprising: a computer readable medium storing computer executable instructions; and one or more hardware processors in communication with the computer readable medium, and configured to execute the computer executable instructions in order to: obtain image data from one or more surveillance image sensors of a satellite; identify, based on the image data, a phenomenon within at least a portion of a field of view of the one or more surveillance image sensors; and generate, based on identifying the phenomenon within the at least the portion of the field of view of the one or more surveillance image sensors, a command to capture an image witha payload image sensor having a field of view smaller than the field of view of the one or more surveillance image sensors.

[0006] Other aspects of the present disclosure are directed to various methods, systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0007] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended figures, in which:

[0009] FIG. 1 depicts a block diagram of an example satellite system according to example embodiments of the present disclosure.

[0010] FIGS. 2A-2B depict an example satellite including sensors according to example aspects of the present disclosure.

[0011] FIG. 3 depicts a block diagram of an example data pipeline according to example aspects of the present disclosure.

[0012] FIG. 4 depicts example image processing according to example aspects of the present disclosure.

[0013] FIG. 5 depicts an example flow diagram of an example method for recomputing targets according to example aspects of the present disclosure.

[0014] FIG. 6 depicts an example maneuvering plan according to example aspects of the present disclosure.

[0015] FIG. 7A-B depicts flowchart diagrams of example methods according to example aspects of the present disclosure.

[0016] FIG. 8 depicts example system components according to example aspects of the present disclosure.DETAILED DESCRIPTION

[0017] Example aspects of the present disclosure are directed to techniques for identifying and / or imaging a phenomenon using one or more satellites. A satellite operator (i.e., a company, government agency, etc.) may utilize imagery satellites to capture images of geographic regions on earth or of imagery directed toward other satellites or celestial bodies. In general, current techniques include a company or individual requesting that satellite images be captured of a specific geographic region, for which a payload sensor may be positioned on board the satellite to capture the desired image of the specific region (e.g., geographic region, orbital region, region of outer space), and return the resulting image to the requesting company or individual.

[0018] Satellite imagery has a broad range of applications for users, including land management, agriculture, environmental protection, disaster response, infrastructure, and defense and intelligence. Some of these uses of satellite imagery function within pre-defined areas of interest on earth. For example, an agricultural user may only be interested in the imagery and associated characteristics of their own land. As such, the agricultural user may request satellite imagery of certain areas on Earth corresponding with their agricultural interest.

[0019] Other intended uses for satellite imagery involve seeking information or characteristics in a larger area, or globally, where they do not have the threshold information to select a specific area of Earth for satellite imagery to be captured. For example, a user may be interested in wildfire monitoring throughout California. In the traditional approach, the user would either have to request satellite images of all of California on a frequent cadence, and analyze each image to look for evidence of wildfire, or would have to make educated guesses on where a fire could be located in the future and hope that when such wildfire occurs that the location and timing of the satellite align to allow that image capture.

[0020] Instead, example embodiments of the present disclosure discuss an alternative approach to the planning and collection process of an imagery satellite. Such approach may include the imagery satellite having capabilities to detect various phenomenon and may further include capturing imagery in the location of such detection. In this way, the satellite operator may collect images of high value for the imagery users without the location and time of interest of such imagery being particularly requested by a specific user.

[0021] Certain phenomena can be of particular interest to various parties. For example, the phenomena can implicate safety concerns or require emergency action. The systems described herein can provide an early warning indication system for such phenomena. The systems described herein can use satellite imaging technology, such as imaging sensors on one or more Low-Earth Orbit (LEO) satellites. A sensor can include a large-view or wide-angle imaging sensor. For example, such imaging sensors can be configured to image large regions of the Earth’s surface or some other field of view relative to the satellite, such as other objects in the same or nearby orbit or even a field of view that includes space objects at a higher altitude or even celestial bodies. The sensor can have a field of view corresponding to a 500 km square region of the Earth’s surface. The sensor can have a field of view corresponding to at least a 1,000 km2region of the Earth’s surface, at least a 5,000 km2region of the Earth’s surface, at least a 10,000 km2region of the Earth’s surface, at least a 25,000 km2region of the Earth’s surface, at least a 75,000 km2region of the Earth’s surface, at least a 100,000 km2region of the Earth’s surface, at least a 250,000 km2region of the Earth’s surface, at least a 500,000 km2region of the Earth’s surface, at least a 1,000,000 km2region of the Earth’s surface, any value therein, or fall within any range having endpoints therein. For example, in some embodiments, a surveillance image sensor can have a field of view corresponding to at least a 250,000 km2region of the Earth’s surface.

[0022] Among other factors contributing to the quality or useability of the captured image, the requirements of the image may include a threshold level of tolerance of what constitutes a phenomenon, which may have high output of certain wavelengths of electromagnetic radiation (e.g., mid-infrared, near-infrared, etc.), may obstruct visibility of portions of the Earth’s surface or other region of interest, or have some other feature distinguishing it from otherwise standard imagery.

[0023] Identifying phenomena may include use of one or more sensors having a particular wavelength range adapted to identify the target phenomenon. For example, when imaging phenomena that have a significant heat signature, a mid-IR sensor may be used. For example, the sensor may be configured to image electromagnetic radiation having a wavelength range of about 3 to 5 microns.

[0024] Various embodiments described herein can be configured to detect and / or analyze infrared (IR) signatures of various phenomena on the Earth’s surface. For example, the systemmay identify large, anomalous IR signatures that may indicate the presence of significant heat sources or other thermal events of interest. For example, such embodiments may be used to monitor environmental changes and / or detect industrial or military activities.

[0025] The systems described herein can include imaging sensors. One or more of the imaging sensors can include one or more processors. Such a distributed processing approach can allow for rapid analysis and response to the detected IR signatures. The processed information from these imaging sensors can be transmitted to a central flight computer, which can integrate the received image data.

[0026] Embodiments disclosed herein can precisely geolocate the detected IR signatures. For example, the systems may identify precise coordinates of the phenomena, and / or associated grid boxes associated therewith, the system can provide spatial information about the location and extent of the detected events. This geolocation data, combined with the time of the image capture, can help identify patterns associated with the phenomena, track changes over time, and / or correlate the observed phenomena with other geographic or temporal information.

[0027] In some embodiments, the systems can include a pixel threshold feature, which allows for the detection of IR signatures that exceed a certain temperature or intensity. This threshold can be adjusted based on the specific application or the expected characteristics of the phenomena being observed. Using appropriate thresholds, the system can filter out background noise and focus on the most significant IR signatures, potentially highlighting relevant events or areas of interest.

[0028] If an IR alert is triggered, the imaging system can up-sample the image to acquire additional, higher-resolution images of the area of interest. This can allow for a more detailed analysis of the detected phenomenon, potentially providing more information about its size, shape, and intensity. The combination of the initial alert and the subsequent up-sampling can enhance the system’s ability to accurately identify and characterize the observed phenomena.

[0029] In some embodiments, the imaging system is configured to operate across one or more ranges of radio frequency (RF) bands, allowing it to capture data from various wavelengths of the electromagnetic spectrum. This flexibility enables the system to detect a broader range of IR signatures, as different phenomena may emit or reflect energy at different frequencies. By using a plurality of RF bands, the system can provide additional detail about the observedphenomena, thereby more accurately characterizing the phenomena and / or more accurately detecting a phenomenon in the first place.

[0030] In some embodiments, the imaging system may include one or more microwave radiometers or bolometers, which can include specialized sensors that can measure the intensity of infrared or microwave / RF radiation with high precision. These radiometers and / or bolometers can complement the imaging data by providing detailed radio or thermal information about the observed phenomena, potentially allowing for more accurate measurements or the identification of subtle non-visual patterns.

[0031] In some embodiments, the system can capture multiple images of the same area over time. By comparing these sequential images, the system can identify and / or filter out any potential artifacts or anomalies that may have been introduced during the image capture or processing stages. Such a multi-image approach can help confirm the veracity of a potentially detected signature, as opposed to a transient or spurious event.

[0032] In some embodiments, the system can include multiple sensors that provide a broader field of view than a single sensor. For example, the system can include 2, 3, 4, 6, 7, 8, or more sensors to create a field of view of at least 90 degrees, 135 degrees, 180 degrees, 270 degrees, or 360 degrees about an axis. In some embodiments, the system may have such field of view about two axes, giving a wide field of view of, for example, both the Earth’s surface and / or outer space.

[0033] In some embodiments, the satellite operator may assign a request to one or more satellites orbiting earth to capture images from the one or more sensors. The one or more satellites may have a trajectory that passes over or near the geographic region associated with the request, such that a payload sensor may be positioned to capture the target image. In some examples, the assigned satellites may have instruments used to calculate or determine that the geographic region associated with the request includes a phenomenon and satisfies a phenomenological threshold indicated by the request. For instance, sensor data processed by the satellite may indicate the presence of a phenomenon. According to example aspects of the present disclosure, satellites may utilize a plurality of sensors, metadata, and / or models to capture images of phenomena that satisfy a threshold, by generating imaging targets that include phenomenon characteristics which satisfy the defined thresholds.

[0034] For example, a phenomenon detection model running on-board a satellite may receive input sensor data, metadata, and / or trajectory data to determine one or more imaging targets where phenomenon coverage is below or above a defined threshold (e.g., a tolerable level / percentage of one or more phenomena in a given geographic area). Sensor data may include images or other data captured by one or more cameras or sensors on board the satellite.

[0035] Example sensors may include a VIS (visible imaging system) sensor, a mid-wave infrared (MWIR) sensor, a longwave infrared (LWIR) sensor, or other types of sensors or cameras. In some examples, the satellite may include a forward-looking surveillance image sensor. The surveillance image sensor may include one or more cameras which are positioned in a forward or backward angle from nadir. Nadir is the point directly below the satellite relative to the Earth. For instance, the surveillance image sensor may be positioned +25 degrees relative to nadir. In some examples, the surveillance image sensor may allow the satellite to receive sensor data indicative of a potential path of travel for the satellite. The +25 degree angle may include a field of view ahead of the satellite such that the satellite may process sensor data within a time frame sufficient to adjust the trajectory or orientation of the satellite. For example, +25 degree angle may be calibrated such that sensor data obtained by the satellite is usable (e.g., not outdated). In some examples, a greater degree angle (e.g., + 50, +60, etc.) may allow for a time frame sufficient to adjust the trajectory of the satellite, but may also allow for conditions to change. For instance, phenomena may form or change position between the time sensor data is captured and processed by the satellite and an updated trajectory reaches a potential imaging target. Additionally, or alternatively, a lesser angle (e.g., +15, +10, etc.) may provide more up-to-date prediction of the location and / or type of the phenomenon, but may not allow for suitable processing time for the satellite to process the sensor data from the surveillance image sensor and make any adjustments to the trajectory of the satellite. The angles relative to nadir used herein are illustrative in nature based on current satellite capability limitations, such as in processing speed, power consumption, and positioning systems. It is contemplated that as such satellite capabilities continue to improve, that an angle of less than +25 degrees may be desirable.

[0036] Satellites may additionally or alternatively utilize metadata for phenomenon detection modeling. The metadata may include data such as the camera temperature, sun angle, Earth surface angle, or slew angle. For instance, the metadata may indicate additional information which may impact the accuracy of the sensor data or trajectory data. The trajectorydata may include a current slew trajectory of the satellite. For instance, the current slew trajectory may include the satellite’s current orientation or movement in reference to its orbit track. In some examples, the current slew trajectory is associated with a path of travel that will pass over geographic regions (e.g., imaging targets) scheduled for image acquisition by the satellite.

[0037] The phenomenon detection model may process the sensor data and metadata to generate imaging targets which satisfy respective thresholds for one or more phenomena. For example, the systems can include a pixel threshold feature, which allows for the detection specific signatures that exceed a certain temperature or intensity. The threshold can be at the pixel level or at a level of a pixel region of the sensor. One or more of these thresholds can be adjusted based on the specific application or the expected characteristics of the phenomena being observed. Using appropriate thresholds, the system can filter out background noise and focus on the most significant signatures, potentially highlighting relevant events or areas of interest. The satellite may generate an updated trajectory to capture imaging targets which are indicative of a phenomenon. For instance, the phenomenon detection model may fuse the sensor data and metadata to allow for more accurate image data that compensates for environmental factors. Environmental factors may distort the sensor data and result in an inaccurate determination of one or more phenomena. For example, sensor data may include a plurality of image frames. In some examples, the phenomenon detection model may perform segmentation to segment the image frames fused with the metadata based on phenomena or other objects detected in the image frames. The phenomenon detection model may detect one or more phenomena in an image frame and segment the image frame to encapsulate the respective phenomena. In some examples, the phenomenon detection model may segment the image frames fused with metadata based on imaging targets.

[0038] For example, the phenomenon detection model can determine one or more candidate pixels, such as a region of candidate pixels. The phenomenon detection model may encapsulate the respective geographic region and further segment based on detected objects such as phenomena. In some embodiments, the phenomenon detection model can compare an intensity of a plurality of candidate pixels to an intensity of a plurality of control pixels. Using this comparison, the phenomenon detection model can determine that the comparison of the intensity of the plurality of candidate pixels to the intensity of the plurality of control pixelssatisfies a phenomenological threshold. Based on the determination that the phenomenological threshold is satisfied, the satellite system can command the payload sensor to capture the image.

[0039] In some examples, the phenomenon detection model may up-sample the segmented image frame. The first up-sampling may occur in a 5 km increment, 10 km increment, 25 km increment, 50 km increment, 100 km increment, 250 km increment, 500 km increment, 1,000 km increment, any increment therebetween, or fall within a range of increments having endpoints therein. For example, in some embodiments the up-sampling is at km increments in the first up-sampling. In some examples, up-sampling the frame can reduce the number of features that are needed to process the image frame by the phenomenon detection model. Additionally or alternatively, up-sampling can allow a user or system to perform further analysis to confirm that one or more candidate pixels correspond to a phenomenon.

[0040] The phenomenon detection model may analyze the up-sampled image segment and determine, based on its features, the presence of one or more phenomena. In some examples, the phenomenon detection model may output one or more imaging targets (e.g., requested imagery of geographic regions) which contain one or more phenomena that satisfy a defined threshold for the one or more phenomena. For instance, the phenomenon detection model may determine a size, intensity, or shape of phenomena in a requested geographic region satisfies a defined threshold and determine an imaging target to cause a satellite system to generate command instructions to update the trajectory of the satellite such that the satellite passes over the geographic region (e.g., imaging target). In some examples, a flight control system of the satellite may determine that the satellite may need to modify a pose of the satellite based on a current trajectory. Additionally or alternatively, the phenomenon detection model may recompute the imaging targets and determine updated imaging targets. In some examples, the updated imaging target may cause satellite system to generate command instructions to update the trajectory of the satellite such that the satellite will pass over the updated imaging targets.

[0041] The technology of the present disclosure may provide several benefits and technical effects. For instance, the technology of the present disclosure may optimize the pose and / or trajectory of satellites tasked to perform satellite imagery by tracking phenomena to capture target satellite imagery. This can avoid the likelihood of missing unforeseen phenomena that are nevertheless target images. Additionally or alternatively, the systems described herein can decrease a number of orbits needed to capture certain target satellite imagery. Accordingly,the technology may increase the flexibility of the satellite by allowing more agile trajectory generation that targets geographic regions which contain one or more phenomena. The technology of the present disclosure may also help to increase the efficiency and management of a satellite system due to the up-sampling of image segments which reduce the number of features the phenomenon detection model must learn to detect and predict one or more phenomena, thereby preserving limited power and computing resources. Moreover, by up-sampling image segments, the technology of the present disclosure may decrease the computing resources required to process sensor data and capture the presence of phenomena.

[0042] The technology of the present disclosure also improves the onboard computing technology of the satellite. For instance, the satellite may include limited power and computing resources to continuously operate the satellite throughout its lifetime. A model running on satellite system may obtain image data from the surveillance image sensor and fuse the image data with the metadata to produce more accurate input data for the model and improve the confidence level of the model. The model may segment the image data fused with the image data and generate an up-sampling to better focus on target image segments. This may include up-sampling features of objects such as phenomena depicted in the image segment. In this way, the model is limited in the number of features it must learn to detect the absence of phenomena in an image frame. Accordingly, the satellite system can avoid wasting computing resources to power the model by more efficiently learning phenomenon features and processing image frames to the detect the absence of phenomena. In this way, the satellite computing system can more efficiently utilize its computing resources.

[0043] With reference now to the FIGS., example embodiments of the present disclosure will be discussed in further detail. FIG. 1 depicts an example satellite system according to example embodiments of the present disclosure. The satellite system 100 may include a number of subsystems and components for performing various operations. For example, the satellite system 100 may include sensors 101, 102, an onboard computing system 103, onboard storage system 105, a flight computing system 107, and a communication system 106. The satellite system 100 may be any computing device which is capable of exchanging data and sharing computing resources. For example, the satellite system 100 may include one or more devices configured to receive, store, or transmit data over physical or wireless technologies. In some examples, the satellite system 100 may include hardware and software. In other examples, thesatellite system 100 may include physical devices connected to one or more networks. In some examples, the satellite system 100 may be any type of satellite (e.g., low earth orbit satellites, medium earth orbit satellites, polar orbit satellites, etc.,) that orbits the Earth and is capable of acquiring imagery.

[0044] The satellite system 100 may include sensors 101, 102. In some examples, the sensors 101, 102 may be cameras. For example, sensor 101 may be a VIS (visible imaging sensor) camera. A VIS camera may include a standard video camera packaged for flight use mounted in a downward oriented manner to provide a continuous view of geographic regions directly below the satellite. Sensor 102 may be a LWIR (long wave infrared) and / or MWIR (mid-wave infrared) camera. A LWIR and / or MWIR camera may include one or more cameras capable of thermal imaging. While examples here describe the sensors 101, 102 as a VIS camera and a LWIR camera, the sensors 101, 102 may include additional or alternative types of cameras such as NIR (near-infrared), SWIR (short-wave infrared), MWIR (mid-wave infrared), etc. In some examples, sensors 101, 102 may be a combination of sensors described herein. While examples here describe the sensors 101, 102 as a VIS camera and / or a LWIR camera, the sensors 101, 102 may include additional or alternative types of cameras such as NIR (near-infrared), SWIR (short-wave infrared), MWIR (mid-wave infrared), or hyperspectral sensors capable of capturing a wide range of wavelengths across the electromagnetic spectrum. In some examples, sensors 101, 102 may also include push broom sensors for line-by-line image acquisition, staring arrays for high-resolution imaging with no moving parts, and / or synthetic aperture radar (SAR) for imaging in all-weather conditions and through obstructions like clouds or vegetation. These sensors may operate individually or in combination to provide enhanced functionality and data collection.

[0045] In some examples, the satellite system 100 may include an onboard computing system 103. For instance, the sensors 101, 102 may be connected to the onboard computing system 103. In some examples, the sensors 101, 102 may be connected via a USB (universal serial bus). In some examples, the sensors 101, 102 may be connected via a MIPI (mobile industry processor interface) interface. For example, a MIPI interface may include a camera module or system which transmits an image from the sensors 101, 102 and stores the image in memory (e.g., storage system 105) as individual frames. In other examples, the sensors 101, 102may be connected to the onboard computing system 103 via any wired or wireless connection such as Bluetooth, SDIO, USB-A, USB-C, etc.

[0046] The onboard computing system 103 may include a number of subsystems and components for performing various operations. For example, the onboard computing system 103 may include a computer vision system 104. The computer vision system 104 may include any computing device which is capable of running computer vision applications. For instance, the computer vision system 104 may include one or more devices configured to perform tasks such as image classification, image segmentation, object detection, etc. In some examples, the computer vision system 104 may include hardware and software. For instance, the computer vision system 104 may include models which process sensor data captured by the sensors 101, 102. An example of the computer vision system 104 processing sensor data is further described with reference to FIGS. 3-4.

[0047] The satellite system 100 may include a storage system 105. For instance, the onboard computing system 103 may be connected to the storage system 105. The storage system 105 may include one or more storage devices for storing data. For example, the storage system 105 may include an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination for storing data. In some examples, the onboard computing system 103 may be connected to the storage system 105 via a wired or wireless connection. For instance, the onboard computing system 103 may be connected to the storage system 105 via an ethernet cable. In some examples, the onboard computing system 103 and storage system 105 may be connected via a Gigabit Ethernet cable. In other examples, the onboard computing system 103 and storage system 105 may be connected over a wireless connection such as Bluetooth, SDIO, etc.

[0048] The onboard computing system 103 may transmit and store data in the storage system 105 over a connection (e.g., ethernet, Gig Ethernet, etc.). By way of example, the sensors 101, 102 may receive sensor data and transmit the sensor data over one or more connections to the onboard computing system 103. In some examples, the computer vision system 104 may perform one or more computer vision tasks and transmit the processed sensor data to the storage system 105 for storage. For instance, the storage system 105 may include data stores such as relational databases, non-relational databases, key-value stores, full-text search engines, messagequeues, etc. In some examples, the computer vision system 104 may store and retrieve data within the storage system 105 over a wired or wireless connection.

[0049] In some examples, the storage system 105 may store data associated with one or more satellites in a satellite constellation. Data associated with the satellites may include a schedule indicative of the pending image acquisition commands / sequences of a given satellite or group of satellites. In some examples, the data associated with satellite may include data indicative of the past, present, and / or future trajectory of the satellite(s). In some examples, the data associated with the satellites may include information associated with the power resources (e.g., power level, etc.), memory resources (e.g., storage availability, etc.), communication resources (e.g., bandwidth), etc. of the satellite(s). In some examples, the data associated with the satellites may include health and maintenance information associated with the satellite(s) (e.g., maintenance schedules, damage reports, other status reports, etc ). In other examples, the data associated with the satellite may include data indicative of the type and / or status of the hardware (e.g., antenna, communication interfaces, etc.) and / or software onboard a satellite (e.g., satellite system 100).

[0050] The satellite system 100 may include a flight computing system 107. For instance, the onboard computing system 103 and flight computing system 107 may transmit data over one or more wired or wireless connections such as Ethernet, Bluetooth, etc. The flight computing system 107 may include different subsystems, software, and hardware for performing various flight control operations. For example, the subsystems may include flight controllers for controlling a motion and / or pose (e.g., orientation) of the satellite system 100. Example flight controllers may include propulsion thrusters, reaction wheels, etc. By way of example, the flight computing system 107 may receive electrical signals over one or more connections from the onboard computing system 103. For instance, the flight computing system 107 may be configured to implement translated controls (e.g., electrical signals) from the onboard computing system 103. In some examples, the flight computing system 107 may implement operations to flight controllers of the satellite system 100 to adjust a trajectory of the satellite.

[0051] In some examples, the flight computing system 107 may include satellite flight software and an execution model to support a communication pathway with the communication system 106. In some examples, the flight computing system 107 may include software to support hardware in a translation layer (e.g., providing a highly efficient packet protocol), a commandinginterface to utilize a low bandwidth channel (e.g., 10s bits / second), an interface to sequence loading, module(s) for attitude control system (ACS) target tracking, a module for image (IMG) captures, module(s) for emergency commanding, module(s) for real time telemetry feedback for critical satellite states module(s) for providing the ability to change pathway settings autonomously based on position (e g., GPS, etc.) and a specific geostationary satellite footprint that has the best line-of-sight (LOS) for the satellite system 100 to encode / decode data transmitted via the communication system 106, etc.

[0052] In some examples, the flight computing system 107 may transmit data over one or more connections to the onboard computing system 103. For instance, data such as health and maintenance information associated with the flight computing system 107 may be transmitted to the onboard computing system 107. In some examples, the data such as the current trajectory of the satellite may be transmitted to the onboard computing system 103. By way of example, the flight computing system 107 may receive electrical signals to adjust a trajectory of the satellite. The flight computing system 107 may determine the satellite has insufficient power / resources to implement the controls or determine, based on its current trajectory, there is insufficient time to implement the controls. In some examples, the flight computing system 107 may transmit response data indicating the current trajectory of the satellite and resources are insufficient to implement controls.

[0053] In some examples, the flight computing system 107 may transmit and store data in the storage system 105 over a connection (e.g., ethernet, Gig Ethernet, etc ). For instance, the flight computing system 107 may be connected to the storage system 105 via a wired or wireless connection. By way of example, the flight computing system 107 may receive one or more electrical signals to control a motion of the satellite. The flight computing system 107 may implement the operations and transmit data over the connection to the storage system 105. For instance, the flight computing system 107 may transmit current trajectory data, an updated trajectory of the satellite after adjusting the trajectory of the satellite, etc. In some examples, the data stored in the storage system 105 may also be accessible by the onboard computing system 103. For instance, the onboard computing system 103 may utilize data (e.g., trajectory data) generated by the flight control system 107 and stored in the storage system 105 for processing sensor data to generate updated trajectories for the satellite. An example of the onboardcomputing system 103 utilizing data generated by the flight computing system 107 to generate updated trajectories is further described with reference to FIG. 5.

[0054] The satellite system 100 may include a communications system 106. The communication system 106 may include hardware and software configured to communicate with remote systems and devices. For instance, the communication system 106 may include antennas that allow the communication system to utilize RT (real time / near real time) communication pathways. RT communication pathways may include a communication pathway via which an image acquisition command is sent directly to the satellite system 100. For instance, a signal may be sent from a ground station to the satellite system 100 when the orbital access and pointing / range requirements of that pathway are met (e.g., when the satellite is in an orbit position to receive a transmission from a ground-based command center). In some examples, the antennas may include an omnidirectional antenna that may be configured to close a link for tasking. In some examples, the communication system 106 may include a phased array antenna (e.g., for higher data rates). In other examples, the communication system 106 may include two separate antennas for a forward and / or reverse link.

[0055] The communication system 106 may include one or more subsystems. The subsystem may include RF (radio frequency) interfaces to transmit (Tx) and / or receive (Rx) via antennas. In some examples, the communication system 106 may utilize full-duplex operation, such that the receiver is enabled at all times. The communication system 106 may utilize frequency separation fl, f2, etc. (e.g., 6 GHz / 4 GHz Tx / Rx frequency separation) for effective isolation between the receiving (Rx) and transmitting (Tx) paths.

[0056] In some examples, the communication system 106 may include GEO communication infrastructure. Example GEO communication infrastructure may include a Very Small Aperture Terminal (VSAT). VSAT may include a two way satellite communication system for communicating with a ground station (e.g., GEO hub(s)) or other satellites such as geostationary satellites. For instance, the communications system 106 may be integrated with a plurality of geostationary satellites with architecture for providing global coverage beams. By way of example, the communications system 106 may establish a network connection for the satellite system 100 by utilizing dedicated bandwidth from geostationary satellites and GEO hub(s). A link may be established to a particular satellite by selecting a corresponding geostationary satellite and GEO hub. As described herein, the GEO hub(s) may be groundstations with communication infrastructure for communication with the geostationary satellites. In some implementations, modems tuned to dedicated frequencies for the entity associated with the communication system 106 may be housed at the GEO hub(s).

[0057] In some examples, the communication system 106 may communicate with ground stations and receive a schedule indicative of the pending image acquisition (e.g., imaging targets) commands. For instance, the ground stations may be associated with a control center which receives requests for image acquisition from one or more third parties, directly from the satellite owner, or satellite operator. In some examples, the ground stations may identify a satellite which is available to acquire the requested imagery. In some examples, the ground station may transmit an acquisition command (e.g., radio signal translation, etc.) to the communication system 106.

[0058] The communication system 106 may be configured to obtain the image acquisition command (e.g., a radio signal translation, etc.) and transmit electrical signals over one or more connections to the flight computing system 107. For instance, the flight computing system 107 may be configured to implement translated controls (e.g., electrical signals) from the communication system 106. In some examples, the flight computing system 107 may implement operations to adjust a trajectory of the satellite and acquire the images. In some examples, the flight computing system 107 may store a schedule of pending image acquisitions (e.g., image requests) in the storage system 105. For instance, the onboard computing system 105 may access and process the schedule of pending image acquisitions. An example of the onboard computing system processing the schedule of pending image acquisitions is further described with reference to FIG. 3.

[0059] In some examples, the communication system 106 may utilize the RT communication pathways to downlink acquired imagery to a ground station. By way of example, the communication system 106 may receive image acquisition commands (e.g., imagery request) from a ground station either directly or through satellite links. The flight computing system 107 may adjust, using flight controllers, the trajectory of the satellite system 100 to navigate to the geographic region specified by the image acquisition command. The sensors 101, 102 may acquire imagery which satisfies the image acquisition command and transmit the imagery to the onboard computing system 103. In some examples, the onboard computing system 103 may store the imagery in the storage system 105. In some examples, the flight computing system 107 may determine that the satellite system 100 is in an orbit position to downlink a transmission to aground station center and the communication system 106 may downlink the requested imagery to the ground station.

[0060] In some examples, the communication system 106 may utilize the RT communication pathway to facilitate the downlink of acquired imagery in near-real-time. For example, the RT communication pathway can use geostationary satellites and / or GEO hubs to establish a persistent communication link between the satellite system 100 and a ground station. The geostationary satellites can relay the imagery from the satellite system 100 to the GEO hub, where the data can be processed and forwarded to the ground station or other designated endpoints via terrestrial networks. This architecture can allow the communication system 106 to overcome latency and orbital constraints associated with standard communication pathways, thereby ensuring faster delivery of imagery data.

[0061] Additionally or alternatively, the communications system 106 may utilize an optical pathway for communication between the satellite system and a ground station. Optical transmissions can be particularly advantageous for high-bandwidth, low-latency applications such as high-resolution imagery downlink. For instance, optical signals, which can operate at frequencies ranging from about 20 GHz to about 40 GHz, may be employed for uplink and / or downlink communications between low Earth orbit (LEO) satellites and ground stations. This optical communication method can provide greater data rates compared to conventional RF communications, which may enable faster transmission of large image files or sensor data.

[0062] In some embodiments, the LEO satellites and / or imaging satellites can incorporate optical communication systems capable of transmitting and receiving signals at specific optical wavelengths. Such optical communications can operate in the Ka-band or other high-frequency ranges (e.g., 26.5 GHz to 40 GHz) to enhance the throughput of image data from the satellite to the ground. The optical pathway may be particularly useful for scenarios where high data volume, such as continuous streaming of imagery or large image files, needs to be transmitted from the satellite system to ground-based stations with minimal latency and high reliability. The use of optical links between the LEO satellites, ground stations, and / or imaging satellites can allow for the implementation of more secure, interference-resistant communication.

[0063] FIG. 2A-2B depicts an example satellite including a surveillance image sensor according to example aspects of the present disclosure. The example satellite 200 may include one or more physical components and instruments for performing various operations. Thesatellite 200 may be low earth orbit satellites, medium earth orbit satellites, polar orbit satellites, or any type of satellite capable of acquiring imagery. For instance, the satellite 200 may include a surveillance image sensor 203 optionally affixed in a forward-oriented position on a surface of the satellite 200. The surveillance image sensor 203 may be VIS camera, MWIR, camera, LWIR camera, or any type of camera that can detect the target phenomenon. In some examples, the surveillance image sensor 203 may be affixed to a front surface of the satellite 200. In other examples, the surveillance image sensor 203 may be affixed to any surface of the satellite 200 which allows for a look ahead angle 202. The look ahead angle 202 may be an angle relative to nadir or an optical axis 201 of a payload sensor of the satellite 200. The surveillance image sensor 203 may have a wide angle of view. For example, the surveillance image sensor 203 may be configured to image at least a 500 km x 500 km region of the Earth’s surface simultaneously. Other configurations are possible.

[0064] By way of example, the satellite 200 may orbit the Earth in an orbit path 205. The satellite 200 may include a payload sensor 204 which captures and transmits imagery of the Earth’s surface. In some examples, the payload sensor 204 may be a camera sensor. The payload sensor 204 may be VIS camera, LWIR camera, or any type of camera. In some embodiments, the payload sensor 204 can include a microradiometer configured to sense radio frequency (RF) waves. In some examples, the payload sensor 204 may be affixed to a downward facing surface of the satellite 200. For instance, the payload sensor 204 may acquire imagery directly below or within a field of view of the satellite 200. In some examples, the payload sensor 204 may acquire imagery which satisfies image acquisition commands. The satellite 200, in its neutral state, may have the payload sensor 204 pointed directly towards the Earth’s surface, such that the payload sensor 204 captures images directly below the satellite 200 in its nadir field of view. The payload sensor 204 capturing imagery directly below the satellite is referred to as nadir. Conversely, the satellite 200 may be maneuverable such that the satellite 200 may have a slew trajectory 206 (i.e. tilt left or right relative to the orbit path 205) such that images may be captured which are not directly below the satellite- referred to as “off-nadir.” As such, by slewing the satellite 200, the optical axis 201 of the payload sensor 204 is moveable such that the field of regard of the payload sensor 204 exceeds the nadir field of view. In order to track a target image (e.g., phenomenon), the satellite 200 may be configured to modify its orientation (e.g., automatically, or in response to a command from a remote computing system), which may be accomplished viaa rotation and / or modified trajectory of the satellite 200 and / or via a physical movement of the payload sensor 204 relative to the satellite 200. This may cause the payload sensor 204 to be turned toward and / or at an identified phenomenon.

[0065] In some examples, the surveillance image sensor 203 may include a look ahead angle 202 of 25 degrees from the optical axis 201. The look ahead angle 202 may include a compound angle. For instance, the look ahead angle may include the slewing angle of the satellite 200 and the datum offset of the boresight of the surveillance image sensor 203. In some examples, the look ahead angle 202 may also include a portion (e.g., one half) of the look ahead sensor 203 field of view (FOV) such that the look ahead angle 202 aligns with the direct (LOS) line of sight of the surveillance image sensor 203. For instance, the surveillance image sensor 203 may include a FOV of 50 degrees. In some examples, the look ahead angle 202 may be a positive angle.

[0066] In some embodiments, the surveillance image sensor 203 can be configured to capture imagery of a target phenomenon. In response, the satellite system 100 can determine a time window in which the payload sensor 204 sensor could image the phenomenon. For example, the time window may be in the future. The determined time window can be about 1 second in the future, about 10 seconds in the future, about 30 seconds in the future, about 1 minute in the future, about 5 minutes in the future, any value therein, or fall within a range having endpoints therein. The satellite system 100 can generate a command for the payload sensor 204 to capture the phenomenon based on the determined time window, for example such that the imagery occurs during that time window. In some examples, the surveillance image sensor 203 may have a larger field of view than the payload sensor 204, such as 2x, 5x, lOx, 20x, or greater field of view compared to the payload sensor 204.

[0067] The satellite 200 may orbit the Earth on a slew trajectory using thrusters or reaction wheels (e.g., flight controllers). A slew trajectory 206 may allow the payload sensor 204 of the satellite 200 to have a moveable optical axis as the satellite 200 orbits the Earth. An example of a slew trajectory is further described with reference to FIG. 6. In some examples, the surveillance image sensor 203 may include a fish-eye field of view (e.g., wide field of view), such as 120 degrees or greater, such that the surveillance image sensor 203 can capture data that is not only within the future field of view of the payload sensor 204 when positioned on nadir, but also to capture imagery within the full future field of regard of the payload sensor whenpositioned off-nadir, thus encompassing the full range of geographic regions capable of being captured by a payload sensor of the satellite 200 within its potential (or capable) slew trajectory 206.

[0068] In some examples, the surveillance image sensor 203 may be configured to identify objects based on one or more spectral bands. For instance, the surveillance image sensor 203 may include a MWIR, a SWIR, and / or a LWIR sensor capable of detecting spectral bands. In some examples, the forward-looking 203 sensor may be configured to detect an object by detecting spectral bands associated with an object. For example, the surveillance image sensor 203 may identify an object as a phenomenon by detecting spectral bands (e.g., 1.36 - 3.08 microns, etc.) which correspond to one or more phenomena. The surveillance image sensor 203 may be configured to capture imagery in a broad or narrow band of wavelengths. The surveillance image sensor 203 may be configured to capture spectral bands of about 1 micron, about 1.5 microns, about 2.0 microns, about 2.5 microns, about 3 microns, about 3.5 microns, about 4.0 microns, about 4.5 microns, about 5 microns, about 6 microns, about 7 microns, about 8 microns, any wavelength value therein, or a range of wavelength bands having endpoints therein. For example, in some embodiments the surveillance image sensor 203 can capture imagery in a wavelength band of between 3 microns and 8 microns. In some embodiments the surveillance image sensor 203 can capture imagery in a wavelength band of between 8 microns and 14 microns, which may correspond to LWIR. Some embodiments may include SWIR that runs from 0.8-2.4 microns. Other examples are possible.

[0069] In some examples, the surveillance image sensor 203 may acquire data indicative of the one or more geographic regions (e.g., imaging targets) ahead of the satellite 200 which have been scheduled for image acquisition. For instance, the surveillance image sensor 203 positioned at a 25 degree look ahead angle 202 may provide data indicative of more optimal imaging targets ahead of the satellite 200 based on the presence of phenomena. For example, one or more phenomena may obscure acquired images. The surveillance image sensor 203 can be positioned at a 25 degree look ahead angle 202 and / or may provide timely data to one or more subsystems (e.g., onboard computing system 103, computer vision system 104. etc.) of the satellite 200 such that the satellite 200 may determine geographic regions near (e.g., ahead of) the satellite 200 which may include one or more phenomena. This may help direct the surveillance image sensor 203 to capture the geographic regions or regions of space whichinclude one or more phenomena. An example of a satellite utilizing sensor data captured by a surveillance image sensor 203 to capture one or more phenomena is further described with reference to FIG. 3. In response to capturing the targets by the surveillance image sensor 203, the payload sensor 204 can be switched on in for the phenomenon to be captured.

[0070] FIG. 3 depicts a block diagram of an example data pipeline according to example aspects of the present disclosure. The following description of dataflow in data pipeline 300 is described with an example implementation in which the satellite system 100 utilizes a phenomenon detection model 302 to process sensor data 304, metadata 305, and trajectory data 306A, 306B to generate output 307 indicative of optimal imaging targets. The example implementation may also include flight controllers 301 which receive output 307 from the phenomenon detection model 302 and generate output 308 to control a motion of the satellite system 100. Additionally, or alternatively, one or more portions of data pipeline 300 may be implemented offboard the satellite system 100.

[0071] The satellite system 100 may include a phenomenon detection model 302 that utilizes the sensor data 304, metadata 305, and trajectory data 306A, 306B to determine imaging targets which include one or more phenomena. In some examples, the phenomenon detection model 302 may be a machine-learned-model. In some examples, the phenomenon detection model 302 may be an analytical model, empirical model, or any combination thereof capable of making a prediction.

[0072] In an embodiment, the phenomenon detection model 302 may be an unsupervised learning model configured to detect, identify, and segment objects depicted in an image frame. In some examples, the phenomenon detection model 302 may include one or more machine-learned models. For example, the phenomenon detection model 302 may include a machine-learned model trained to detect objects in a specific context (e.g., phenomena over bodies of water, over land, etc.). In some examples, the phenomenon detection model 302 may include a machine-learned model trained to distinguish phenomena from other objects such as snow, ice, or ocean foam by executing segmentation techniques.

[0073] The phenomenon detection model 302 may be or may otherwise include various machine-learned models such as, for example, regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbors models, Bayesian networks, or other types of modelsincluding linear models or non-linear models. Example neural networks include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks.

[0074] The phenomenon detection model 302 may be trained through the use of one or more model trainers and training data. The model trainers may be trained using one or more training or learning algorithms. One example training technique is backwards propagation of errors. In some examples, simulations may be implemented for obtaining the training data or for implementing the model trainer(s) for training or testing the model(s). In some examples, the model trainer(s) may perform supervised training techniques using labeled training data. As further described herein, the training data may include labelled image frames that have labels indicating one or more phenomena and the segmentation of phenomena (e.g., isolated phenomena, storm / weather systems, etc.). In some examples, the training data may include simulated training data (e.g., training data obtained from simulated scenarios, inputs, configurations, previous satellite orbits, etc.).

[0075] Additionally, or alternatively, the model trainer(s) may perform unsupervised training techniques using unlabeled training data. By way of example, the model trainer(s) may train one or more components of a machine-learned model to execute phenomenon capture through unsupervised training techniques using an objective function (e.g., costs, rewards, heuristics, constraints, etc.). In some implementations, the model trainer(s) may perform a number of generalization techniques to improve the generalization capability of the model(s) being trained. Generalization techniques include weight decays, dropouts, or other techniques.

[0076] The phenomenon detection model 302 may obtain sensor data 304 from one or more sensors 101,102 of the satellite system 100. In some examples, the sensor data 304 may include image data. In some examples, the sensor data 304 may include video data. In some examples, the sensor data 304 may include image data captured by the surveillance image sensor 203. For instance, the phenomenon detection model 302 may obtain sensor data 304 including image data from the surveillance image sensor 203 depicting geographic regions ahead of the satellite system 100. For example, the satellite system 100 may be traveling along a current trajectory and the surveillance image sensor 203 may acquire sensor data 304 (e.g., image data, video data, etc.,) depicting geographic regions in the path of travel of the satellite 200 (e.g., satellite system 100).

[0077] In some examples, the sensor data 304 may include one or more image frames. For instance, the current trajectory data 306A of the satellite (e.g., satellite system 100) may be associated with scheduled image acquisitions. The scheduled image acquisitions may indicate requests for imagery of a specific geographic region (e.g., imaging targets). In some examples, the surveillance image sensor 203 may obtain sensor data 304 including one or more image frames depicting imaging targets scheduled for image acquisition ahead of the satellite system 100. By way of example, the satellite system 100 may be traveling along a current trajectory and the surveillance image sensor 203 may obtain sensor data 304 (e.g., one or more image frames) depicting imaging targets scheduled for image acquisition.

[0078] The phenomenon detection model 302 may access metadata 305 associated with one or more environmental conditions. Metadata 305 may include additional data which improves the accuracy of sensor data 304 or confidence level of the phenomenon detection model 302. For example, metadata 305 may include map data of historical phenomenon location, monthly snow maps, land cover classifications, camera temperatures, sun angle, time of day, or altitude. Metadata may include any data which may be used to increase the accuracy of sensor data 304 or improve the confidence level of the phenomenon detection model 302. By way of example, sensor data 304 may include one or more image frames of a geographic region which depicts phenomenon-like formations. Metadata 305 including a map of historical phenomenon location may indicate a consistent pattern of one or more phenomena over the geographic region. In some examples, the metadata 305 including the historical phenomenon locations may improve a confidence level determination of the phenomenon detection model 302 that the phenomenonlike formation depicted in the sensor data 304 includes one or more phenomena.

[0079] In some examples, the phenomenon detection model 302 may fuse the sensor data 304 and metadata 305. For instance, the phenomenon detection model 302 may be trained to detect the presence of phenomena in an image frame. In some examples, the phenomenon detection model 302 may fuse the sensor data 304 and metadata 305 to identify additional features within the image frames which may increase the accuracy of the sensor data 304. In some examples, fusing the sensor data 304 and metadata 305 may compensate for environmental factors which may result in inaccurate identification of phenomena. For instance, fusing the sensor data 304 and metadata 305 may help improve focal loss.

[0080] Metadata 305 may include any data which may adversely impact sensor data 304 or the sensors 101, 102. For instance, metadata 305 may include the camera (e.g., sensors 101, 102) temperature, sun angle, Earth surface angle, and slew angle may be fused with sensor data 304 to improve the accuracy of the sensor data 304. Sensor temperature metadata may include the temperature inside the satellite itself, the temperature of the geographic region detected by the sensor 101, 102, or any combination thereof. By way of example, sensor data 304 including an image frame of a geographic region may depict a bright phenomenon-like formation. In an embodiment, the phenomenon detection model 302 may fuse sensor data 304 with sensor temperature metadata indicating temperatures of the geographic region are below freezing. For instance, the sensor 102 may be a MWIR sensor capable of thermal imaging. The phenomenon detection model 302 may identify the bright phenomenon-like formation as snow or ice rather than a phenomenon based on the sensor temperature metadata. In another example, sensor temperature metadata may include a temperature of the payload sensor 204. For instance, the payload sensor may become hot causing adverse impacts to image quality.

[0081] To avoid false positives, the phenomenon detection model 302 may be configured to identify artifacts within the sensor data 304. One approach to doing this is to compare multiple images, such as images close in time, to determine whether an artifact is present. For example, the phenomenon detection model 302 may obtain first and second sensor data 304 from the surveillance image sensor 203. The phenomenon detection model 302 may compare the first sensor data 304 to the second sensor data 304. Based on comparing the first sensor data 304 to the second sensor data 304, the phenomenon detection model 302 can determining that an identified phenomenon is not associated with an artifact.

[0082] In an embodiment, the phenomenon detection model 302 may fuse sensor data 304 with sun angle metadata, Earth surface metadata, and slew angle metadata. For instance, the flight computing system 107 may determine, based on its orbit position, the sun angle, Earth surface angle, and slew angle relative to the surveillance image sensor 203. In some examples, the angle at which the image frame was captured may result in varying brightness, exposure, shadows, density, contrast, saturation, etc. The phenomenon detection model 302 may more accurately identify the bright phenomenon-like formation as a target phenomenon rather than a reflection of light from land or other objects based on the sun angle metadata, Earth surface metadata, and slew angle metadata.

[0083] The phenomenon detection model 302 may access current trajectory data 306A indicating a current trajectory of the satellite (e.g., satellite system 100). In some examples, current trajectory data 306A may include the current speed, orbital position, or planned waypoints for the satellite. In some examples, the current trajectory data 306A may be transmitted over one or more connections from the flight computing system 107. In some examples, the current trajectory data 306 A may be accessed from the storage system 105. For instance, the flight computing system 107 may actively store current trajectory data 306A in the storage system 105 where the onboard computing system 103 may access.

[0084] In some examples, the current trajectory data 306A may be associated with imaging targets (e.g., geographic regions). For instance, the current trajectory data 306A may indicate an orbit path or waypoint which passes over imaging targets scheduled for image acquisition. In some examples, the phenomenon detection model 302 may associate image frames included in sensor data 304 with geographic regions (e.g., imaging targets) which are scheduled for image acquisition. For instance, the phenomenon detection model 302 may fuse sensor data 304 and metadata 305 associated with imaging targets to determine whether the imaging targets scheduled for image acquisition include one or more phenomena.

[0085] In some examples, the phenomenon detection model 302 may fuse the sensor data 304 and metadata 305 and process the image frames to determine one or more phenomena associated with the imaging targets scheduled for image acquisition. For instance, as shown in FIG. 4 the phenomenon detection model 302 may receive an input image 401. Input image 401 may be sensor data 304 captured by a surveillance image sensor 203 of the satellite. In some examples, the input image 401 may include an image frame fused with metadata 305. In some examples, the phenomenon detection model 302 may perform phenomenon segmentation or other determination technique, such as candidate pixel determination 402 techniques to segment the input image 401 based on the presence of phenomena depicted in the image frame. In some examples, the phenomenon detection model 302 may perform candidate pixel determination 402 techniques to segment the input image 401 based on imaging targets. For instance, the input image 401 may include a plurality of imaging targets and the input image 401 may be segmented based on the respective imaging targets to determine one or more candidate pixels that may correspond to one or more phenomena. In some examples, the segmented image frames mayinclude one or more objects (e.g., phenomena). For instance, the segmented image frames may be analyzed to capture one or more phenomena.

[0086] Candidate pixel determination 402 techniques may include analyzing the sensor data 304 (e.g., input image frame 401) fused with the metadata 305 and projecting a candidate pixel region boundary 416 on the image frame. The candidate pixel region boundary 416 can form a bounding shape that separates a control pixel region 408 from a candidate pixel region 412. A bounding shape may be any shape (e.g., polygon) that includes one or more imaging targets. Additionally, or alternatively, a bounding shape may include a shape that matches the outermost boundaries and contours of those boundaries for an imaging target. The bounding shape may bound the one or more candidate pixels. One of ordinary skill in the art will understand that other shapes may be used such as circles, squares, rectangles, etc. In some examples, the bounding shape may be generated on a per-pixel level.

[0087] The phenomenon detection model 302 may perform candidate pixel determination 402 to identify pixels of a candidate pixel region 412 that represent or correspond to a phenomenon within the input image 401 based on detecting one or more imaging targets.Additionally or alternatively, the phenomenon detection model 302 can generate an up-sampling 403 of the input image 401 (e.g., image segments). The up-sampling 403 may include a filter encapsulating or otherwise focusing on the image segment. In some examples, the up-sampling 403 may improve the clarity or sharpness or other detail of the input image 401 and / or of the candidate pixel region 412.

[0088] In some embodiments, the phenomenon detection model 302 can determine a set of coordinates corresponding to or indicative of a boundary of the field of view of the surveillance image sensor 203. The phenomenon detection model 302 may generate a grid or other ordered arrangement within the boundary of the field of view of the surveillance image sensor 203. The ordered arrangement (e.g., grid) can include a plurality of cells or other subparts of the ordered arrangement. The phenomenon detection model 302 may identify the phenomenon within the field of view of the surveillance image sensor 203 by determining a location of the phenomenon within one or more of the subparts of the ordered arrangement. For example, the phenomenon detection model 302 may identify the phenomenon within one or more cells of a grid of cells. The phenomenon detection model 302 may use these subparts to up-sample the ordered arrangement (e.g., grid) to generate a second ordered arrangement having a plurality ofup-sampled subparts. In some embodiments, the phenomenon detection model 302 can determine a location of the phenomenon within one or more of up-sampled subparts.

[0089] For example, in some examples, the phenomenon detection model 302 may generate an up-sampling 403 of the candidate pixel region 412 to allow the satellite system 100 better identify the phenomena within the candidate pixel region 412. The up-sampled candidate pixel region 412 can result in an up-sampled candidate pixel region 420. Additionally or alternatively, the up-sampled candidate pixel region 420 can allow a user to more easily identify one or more features of the one or more phenomena and / or update a shape of the candidate pixel region boundary 416. In some examples, the phenomenon detection model 302 may generate the up-sampled candidate pixel region 420 to magnify the entire candidate pixel region 412.Alternatively, the up-sampling 403 may magnify a portion of the candidate pixel region 412, such as a 5 km (kilometer) portion, of the image segment (e.g., centered on the candidate pixel region 412). In some examples, the phenomenon detection model 302 may perform candidate pixel determination 402 to identify one or more phenomena within smaller image segments (e.g., 5 km image segments) of the input image 401. The smaller image segments may correspond to a grid of image segments of the input image 401 that can be used in determining the candidate pixel region 412. In some examples, the up-sampling 403 may be done at any interval, such as 10 km, 20 km, etc. While examples here describe the up-sampling 403 as a post-processing operation, the phenomenon detection model 302 may also generate the up-sampling 403 during candidate pixel determination 402 as a single operation.

[0090] In some embodiments, the satellite system 100 can compare an intensity of one or more pixels of the candidate pixel region 412 to an intensity of one or more pixels of the control pixel region 408. Using this comparison, the satellite system 100 can determine that the comparison of the intensity of the one or more pixels of the candidate pixel region 412 with the intensity of the one or more pixels of the control pixel region 408 satisfies a phenomenological threshold. Based on the determination that the phenomenological threshold is satisfied, the satellite system 100 can command the payload sensor 204 to capture the image. The phenomenological threshold may correspond to a number of pixels having captured at least a threshold value (e.g., intensity, duration, wavelength, etc.) of light. Additionally or alternatively, the phenomenological threshold can correspond to a threshold value of the light across one or more pixels.

[0091] The phenomenon detection model 302 may analyze the up-sampled image segments and determine one or more characteristics indicative of one or more phenomena. For instance, the phenomenon detection model 302 may generate characteristics data (e.g., labels) that correspond to the characteristics of the bounding shape. Labels may include the classification of objects (e.g., land, water, etc.), the type of objects (e.g., phenomena, snow, ocean foam, etc.), density, etc. In some examples, the characteristics data (e.g., labels) may indicate the presence of phenomena in the up-sampled candidate pixel region 420. In some examples, the characteristics data may indicate the absence of phenomena in the up-sampled image segment. In some examples, the up-sampling 403 may improve the characteristics data used to identify one or more phenomena or the absence of phenomena.

[0092] By way of example, the phenomenon detection model 302 may analyze the up-sampled image segments and label up-sampled objects within the up-sampled image frame based on fused metadata 305. For instance, the phenomenon detection model 302 may label the candidate pixel region boundary 416 which can include an up-sampled red, orange, and / or yellow phenomenon-like object as a phenomenon due to sensor temperature metadata indicating the temperature of the up-sampled phenomenon-like object matches a temperature range typical of a phenomenon (e.g., phenomenon characteristic) in the geographic region. In some examples, the phenomenon detection model 302 may determine the up-sampled phenomenon-like object is a phenomenon and generate a phenomenon label to indicate the presence of phenomena.

[0093] In some examples, the phenomenon detection model 302 may determine an absence of phenomena within the up-sampled image frame based on fused metadata 305. By way of example, the phenomenon detection model 302 may analyze an up-sampled image segment and label a bounding shape which includes a phenomenon-like object as phenomenon-less due to sun angle metadata and slew angle metadata which indicates the phenomenon-like object appearance is a result of sunlight reflecting off an Earth or other surface. In some examples, phenomenon detection model 302 may determine that the up-sampled image frame does not include one or more phenomena and generate a phenomenon label to indicate the presence of phenomena.

[0094] In other examples, the phenomenon detection model 302 may determine a level of phenomenon coverage depicted in an up-sampled image frame and generate a label indicating the level of phenomenon coverage in the image frame. For instance, the phenomenon detectionmodel 302 may generate a phenomena-coverage label. In some examples, the phenomena-coverage label may be associated with a phenomenon label. In other examples, the phenomena-coverage label may be nested within a phenomenon label. The phenomena-coverage may be a percentage, ratio, or any measure of one or more phenomena relative to the up-sampled image frame (e.g., geographic target). By way of example, the phenomenon detection model 302 may analyze an up-sampled image segment and detect the presence of phenomena. In some examples, the phenomenon detection model 302 may determine a level of phenomena within the up-sampled image frame. Determining a level of phenomena may include determining the percentage of pixels associated with the detected phenomenon relative to the candidate pixel region boundary 416 within the up-sampled image segment. The percentage of pixels may correspond to a percentage of pixels that exceed a minimum temperature threshold or other threshold described herein. In some examples, the phenomenon detection model 302 may determine based on metadata 305 the level of one or more phenomena within the up-sampled image frame. For instance, Earth surface angle metadata may indicate that the detected phenomena are not dense enough to satisfy a target phenomenon threshold. In some examples, the level of one or more phenomena may be determined based on metadata 305 indicating the density of phenomena depicted in the up-sampled image segment.

[0095] In some examples, the phenomenon detection model 302 may detect phenomena within the up-sampled image segment and generate a phenomena-coverage label to indicate the level of one or more phenomena. The phenomena-coverage label may include an integer value, percentage value, ratio, or any value which indicates a consistent measure of one or more phenomena.

[0096] Returning to FIG. 3, the phenomenon detection model 302 may determine based on current trajectory data 306A and sensor data 304 (e.g., image data) fused with metadata 305, one or more imaging targets and one or more phenomena associated with the imaging target. For instance, the phenomenon detection model 302 may associate sensor data 304 (e.g., input image 401) with an imaging target based on current trajectory data 306A. The phenomenon detection model 302 may determine based on labeled up-sampled image segments the presence of phenomena and the percentage of coverage (e.g., pixel coverage) of one or more phenomena within the up-sampling 403 and / or the candidate pixel determination 402.

[0097] In some examples, the phenomenon detection model 302 may determine a comparison between the one or more phenomena associated with an imaging target and a threshold level of one or more phenomena. For instance, the current trajectory data 306A may include data associated with a phenomenon threshold or tolerance level for respective imaging targets. For example, the current trajectory data 306A, may indicate imaging targets scheduled for image acquisition.

[0098] In some examples, the imaging targets scheduled for image acquisition may indicate additional acquisition parameters. For instance, the imaging targets may indicate a phenomenon threshold or tolerance level to satisfy the acquisition request. By way of example, a phenomenon threshold or tolerance level of 30% may require that the acquired image contain at least 30% of one or more phenomena to satisfy the image acquisition request. In some examples, the additional acquisition parameters may include a cost associated with the image acquisition. A cost may include commercial factors such as failure of previous attempts to acquire imagery, client value (e.g., priority of clients), or any other strategic prioritization rationale.

[0099] For instance, the phenomenon detection model 302 may determine a failure cost acquisition parameter associated with the image acquisition. The failure cost may include a generated integer or percentage value representing previous attempts associated with the image acquisition. In some examples, the failure cost may be any quantified representation of a previous failure associated with the image acquisition of a phenomenon within a geographic region.

[0100] In some examples, the phenomenon detection model 302 may determine a client value acquisition parameter. The client value acquisition parameter may include a priority of clients (e.g., large clients, medium clients, small clients, etc.). For instance, large clients may include clients which generate large quantities of image acquisition requests. In some examples, large clients may be associated with a high priority acquisition parameter to indicate a higher priority for image acquisition requests associated with the large client. In other examples, small or medium clients may be associated with a lower priority acquisition parameters to indicate the lower priority of image acquisition requests associated with the small or medium clients.

[0101] In other examples, the image acquisition parameters may include other relationship considerations. For instance, the image acquisition parameter may include strategic reasons for prioritizing an image acquisition request. For example, the image acquisition requestsmay be associated with a new client. Tn some examples, new clients may be associated with a high priority image acquisition parameter to display an ability to satisfy future image acquisition requests for the new client. The image acquisition parameters may include any other strategic reason for prioritizing an image acquisition request.

[0102] In some examples, geography may also be considered. Geographic regions which are more difficult to acquire imagery satisfying a phenomenon threshold or tolerance level may be associated with a higher cost than other geographic regions due to the increased difficulty in capturing the phenomena. In addition, a priority may be placed on certain geographic regions that rarely satisfy a phenomenon threshold or tolerance level, and thus may be prioritized in circumstances where the phenomenon threshold is met.

[0103] In other examples, the additional acquisition parameters may include a time parameter. The time parameter may indicate that the requested image must be acquired within a specified time period. By way of example, the time parameter may indicate that imagery be acquired within 72 hours from request.

[0104] The phenomenon threshold or tolerance level may include an integer value, percentage value, or any value which indicates a consistent measure of one or more phenomena. In some examples, the phenomenon threshold or tolerance level may include an upper limit threshold, lower limit threshold or an exact threshold or tolerance level. In some examples, satisfying the threshold or tolerance level may include phenomena-coverage above the phenomenon threshold, below the phenomenon threshold or exactly matching the phenomenon threshold.

[0105] The phenomenon detection model 302 may compare the phenomena-coverage label of the up-sampled image segment to the phenomenon threshold or tolerance level for the requested imagery of the geographic region and determine imaging targets which satisfy the phenomenon threshold acquisition parameter. By way of example, the phenomenon detection model 302 may receive sensor data (e.g., input image 401), perform candidate pixel determination 402 to segment the input image 401, generate an up-sampling 403 to enhance or magnify the image segment, and / or label the up-sampled image segment. In some examples, the phenomenon detection model 302 may identify a first phenomenon and a second phenomenon within a first and second geographic region (e.g., imaging targets) scheduled for image acquisition based on labeling up-sampled image segments. In some examples, the phenomenondetection model 302 may determine the phenomena-coverage associated with the first and second imaging targets within the up-sampled image segments based on labeling the up-sampled candidate pixel region 420. The phenomenon detection model 302 determine based on current trajectory data 306A that the first imaging target includes a phenomenon threshold acquisition parameter of 10% and the second imaging target includes a phenomenon threshold acquisition parameter of 40%. The phenomenon detection model 302 may compare the phenomenon level labels for the respective imaging targets and determine the first imaging target should be captured due to a phenomenon label indicating 80% of one or more phenomena and a phenomenon label indicating 85% of one or more phenomena for the second imaging target.

[0106] In some examples, the phenomenon detection model 302 may determine both first and second imaging targets satisfy their respective phenomenon threshold acquisition parameters and prioritize imaging targets based on other acquisition parameters. For instance, an imaging target may be associated with a time parameter that will be exceeded if the image is not acquired during the current orbit. In some examples, a cost acquisition parameter may influence priority of imaging targets which satisfy phenomenon threshold acquisition parameters. For instance, a first image acquisition may be valued higher than second image acquisition based on commercial factors (e.g., failure, client value, etc.) and may increase the priority of the imaging target and the phenomenon detection model 302 may determine the first imaging target should be acquired instead of the second imaging target.

[0107] The phenomenon detection model 302 may determine imaging targets which satisfy the phenomenon threshold acquisition parameters and transmit the imaging targets to the flight computing system 107 to generate an updated trajectory. For instance, the current trajectory data 306A may indicate that the current trajectory of the satellite (e.g., satellite system 100) will pass over the imaging targets which satisfy the phenomenon threshold acquisition parameters (e.g., to capture target phenomenon). The phenomenon detection model 302 may generate output 307 indicative of the imaging targets that satisfy the phenomenon threshold acquisition parameters. In some examples, the phenomenon detection model 302 may transmit output 307 to the flight controller 301 of the flight computing system 107. In some examples the flight controllers 301 may be configured to receive the output 307. For instance, the output 307 may include one or more electrical signals.

[0108] In some examples, the satellite system (e.g., satellite system 100) may include a normal image tasking queue, which can prioritize imaging tasks based on one or more factors such as customer requests, task priority levels, and / or previously failed acquisitions. A forwardlooking sensor (e.g., surveillance image sensor 203) may detect a phenomenon of interest within its field of view, and the phenomenon detection model 302 may determine that the detected phenomenon satisfies the phenomenon threshold acquisition parameters. In response, the phenomenon detection model 302 may reprioritize the existing imaging tasks by accelerating the new phenomenon-based imaging request higher in (e g., to the top of) the queue. The flight computing system 107, in conjunction with the flight controller 301, may then update the satellite’s trajectory and imaging schedule to prioritize the phenomenon-based imaging request over other tasks. For instance, the flight controller 301 may adjust the satellite’s orientation, sensor activation, and / or timing to ensure the detected phenomenon is captured, while deferring or rescheduling the lower-priority imaging tasks in the queue. This dynamic task reprioritization ensures that transient or high-value phenomena are not missed, enhancing the operational utility of the satellite system.

[0109] By way of example, the flight controllers 107 of the flight computing system 107 may receive output 307 (e.g., electrical signals) over one or more connections from the phenomenon detection model 302 of the computer vision system 104. For instance, the flight computing system may be configured to implement translated controls (e.g., electrical signals) from the computer vision system 103. In some examples, the flight computing system 107 may be configured to receive output 307 indicative of imaging targets and determine an updated trajectory. For instance, the current trajectory of the satellite system 100 may not include a way point or slew trajectory which passes over the imaging target indicated by the output 307. The flight computing system 107 may implement the output 307 by computing an updated trajectory and implementing operations to flight controllers 301. For instance, the flight controllers 301 may generate output 308 to control a motion of the satellite system 100. For example, the output 308 may include one or more command instructions to control a motion and / or pose of the satellite based on the updated trajectory. For example, the motion may include a rotation of the satellite to modify the pose of the satellite. In some embodiments, the one or more command instructions can be transmitted to a second satellite to cause a pose of the sensor of that second satellite to be configured (e.g., modified) to capture the one or more phenomena.

[0110] In some examples, the flight computing system 107 may determine the output 307 received from the phenomenon detection model 302 cannot be executed. For instance, the flight computing system 107 may determine, based on the current position, orbital speed, slew angle, etc. that navigating to an imaging target indicated by the output 307 results in a low probability of reaching the imaging target. For instance, the satellite system 100 may orbit the Earth at thousands of miles per hour and updating the trajectory may require sufficient time to maneuver without missing the imaging target. In some examples, the flight computing system 107 may determine the output 307 indicative of an updated trajectory is not possible or has a low probability of passing over the imaging target and generate updated trajectory data 306B accessible by the phenomenon detection model 302 to recompute the output 307 (e.g., imaging targets). Passing over the imaging target may include aligning the payload sensor 204 with the imaging target (e.g., phenomenon). In some examples, passing over the imaging target may include aligning the payload sensor 204 with a field of view (e.g., off-nadir) of the imaging target. An example, of the phenomenon detection model 302 recomputing output 307 is further 34 described with reference to FIG. 5

[0111] In some examples, the flight computing system 107 may determine the output 307 indicative of an updated trajectory is not possible or has a low probability of acquiring off-nadir imagery and generate updated trajectory data 306B accessible by the phenomenon detection model 302 to recompute the output 307. For instance, the payload sensor 204 may acquire imagery which is not directly below the satellite 200 as the satellite 200 slews. In some examples, the imaging target may be outside of the field (e.g., off-nadir) of view of payload sensor 204. For instance, sensor data 304 captured by the surveillance image sensor 203 may include the geographic regions in the field of view of the payload sensor 204 based on the current position, orbital speed, slew angle, etc. of the satellite 200 (e.g., satellite system 100). As the satellite 200 orbits the earth the position, orbital speed, slew angle, etc. may change such that the output 307 indicative of an updated trajectory is not possible or has a low probability of acquiring off-nadir imagery.

[0112] FIG. 5 depicts a flow diagram of an example method 500 for recomputing targets according to example aspects of the present disclosure. One or more portion(s) of the method 500 may be implemented by a computing system that includes one or more computing devices such as, for example, the computing systems described with reference to the other figures (e.g., asatellite system 100, onboard computing system 103, computer vision system 104, flight computing system 107, etc.). Each respective portion of the method 500 may be performed by any (or any combination) of one or more computing devices. Moreover, one or more portion(s) of the method 500 may be implemented as an algorithm on the hardware components of the device(s) described herein.

[0113] At (505), the sensors 101, 102 of the satellite system 100 may capture sensor data 304 (e.g., input image 401). In some examples, the sensor data 304 may be captured by the surveillance image sensor 203 of the satellite. In some examples, the sensor data 304 may include a forward-looking image (FLI) of the geographic regions ahead of the satellite. In some embodiments, the sensor data 304 for the FLI may be from a first satellite and the surveillance image sensor 203 may be included onboard a second satellite. The FLI may include sensor data 304 depicting geographic regions ahead of the satellite 200.

[0114] At (510), the phenomenon detection model 302 may perform candidate pixel determination 402 to segment the input image 401 and identify one or more phenomena. For instance, the phenomenon detection model 302 may segment the input image 401 based on objects (e.g., phenomenon-like objects) detected in the input image, imaging targets, etc. In some examples, the phenomenon detection model 302 may up-sample the image segments and label the up-sampled image segments to identify object characteristics (e.g., phenomenon characteristics). For instance, the phenomenon detection model 302 may label the up-sampled image segments as phenomena and generate labels indicative of the level of one or more phenomena within the up-sampled image segment.

[0115] The phenomenon detection model 302 may determine, based on comparing the level of one or more phenomena for the respective image frame (e.g., imaging target) to a phenomenon threshold acquisition parameter, imaging targets which satisfy the phenomenon threshold acquisition parameter. In some examples, the phenomenon detection model 302 may output 307 data indicating the imaging targets which satisfy the phenomenon threshold acquisition parameter and transmit the output to the flight computing system 107.

[0116] At (515-520), the flight computing system 107 may receive the output 307 from the phenomenon detection model 302 and compute a trajectory of the satellite 200 which includes the imaging targets defined within the output 307. In some examples, the flight computing system 107 may implement the output 307 operations to flight controllers 301. Forinstance, the flight controllers 301 may determine an updated trajectory and generate output 308 to control a motion of the satellite system 100. In some examples, the output 308 may include one or more command instructions to control a motion of the satellite 200 based on the updated trajectory. In some examples, the command instructions may cause the satellite 200 to travel along the updated trajectory.

[0117] At (525), the flight controllers 301 may be unable to adjust the trajectory of the satellite 200 to travel along the updated trajectory. The trajectory may include the current orientation and direction of the satellite. In some examples, the slew trajectory 206 may be a subset of the trajectory of the satellite. For instance, the slew trajectory 206 may include a slew angle or position relative to the overall trajectory (e.g., orientation, direction, position, etc.) of the satellite as it travels along an orbit path 205. For instance, the probability of the satellite 200 passing over the imaging target indicated by the phenomenon detection model 302 may be low based on the current position of the satellite 200 orientation, orbital speed, slew position or angle, etc. In some examples, the flight controllers 301 may not have sufficient power or resources to execute the computed trajectory. For instance, thruster or reaction wheels may require additional power to align the satellites with the computed trajectory quickly enough to pass over determined imaging targets.

[0118] In some examples, the probability of imminently passing over one or more phenomena may be calculated by the flight computing system 107. For instance, the flight computing system 107 may include different subsystems, software, and hardware for performing various flight control operations. In some examples, the probability of an imminently passing over the phenomena may be calculated using orbital speed equations, acceleration equations, orbital period equations, etc. In some examples, the resulting calculations may produce probability of the satellite 200 passing over the imaging target. In some examples, a low probability of imminently passing over may be determined if the calculated probability is below 50%. In some examples, a low probability of imminently passing over may be determined if the calculated probability is below 70%. In some examples, any probability percentage may be considered a low probability. In other examples, a low probability of imminently passing over may be determined based on additional factors such as image acquisition parameters, priority of scheduled image acquisitions, or availability of other satellites to acquire the requested imagery.

[0119] By way of example, the flight computing system 107 may determine a low probability of imminently passing over by calculating a 60% chance of passing over the imaging target. In some examples, the flight computing system may transmit via the communications system 106 data indicating the low probability of imminent overpass to a ground station and a second satellite may be assigned to acquire the imagery which includes an 80% probability of imminent overpass.

[0120] At (530), the flight computing system 107 may determine a low probability of passing over the determined imaging targets and transmit data to the onboard computing system 103 to execute steps (510-520). In some examples, the flight computing system 107 may transmit data to the onboard computing system 103 to cause the surveillance image sensor 203 to capture additional sensor data 304. For instance, the satellite (e.g., the satellite 200) may be orbiting at thousands of miles per hour and an updated input image 401 may be needed to recompute the trajectory of the satellite.

[0121] At (535), the flight computing system 107 may determine a high probability of imminent overpass and generate one or more command instructions to control a motion and / or pose of the satellite and / or of the surveillance image sensor 203 based on the updated trajectory. A high probability may include 70% or greater, 60% or greater, or any reasonable percentage. The satellite may pass over the determined imaging target and utilize one or more sensors 101, 102 to acquire imagery of the respective geographic region. In some examples, the satellite system 100 may downlink the acquired imagery to a ground station at a point during orbit where the communication system 106 may transmit the acquired imagery.

[0122] FIG. 6 depicts an example satellite maneuvering plan according to example aspects of the present disclosure. The example maneuvering plan includes an orbit track 600 which illustrates the location of a satellite (e.g., satellite system 100, satellite 200, etc.) as it orbits the Earth. In some examples, the orbit track 600 may indicate a complete path around the earth. In some examples, the orbit track 600 may prevent satellites from colliding. For instance, the orbit track 600 may indicate a unique path of travel for the satellite which allows for the coordination or orchestrion of other satellites orbiting the Earth. For example, a plurality of satellites in a constellation may simultaneously orbit the Earth on various orbit tracks 600 which do not intersect or interfere with other satellite orbits. In other examples, the orbit track 600 maybe shared by multiple satellites such that the satellites are in orbit positions spaced on the same on the orbit track 600 to avoid collisions.

[0123] Satellites having maneuvering capabilities may be capable of “slewing” as the satellite progresses along the orbit track 600. For example, while the satellite may maintain its position on the orbit track 600, it may slew left or right, relative to its direction of travel, such that geographic areas of Earth may be captured (i.e., by the payload sensor 204) which are not directly below the position of the satellite on the orbit track 600. While slewing allows a satellite to capture imagery which is not directly below its orbit track 600, greater amounts of skewing alter the perspective of the captured imagery and may be undesirable beyond a certain slew angle 206. Therefore, a predefined upper edge 601 and lower edge 603 may be established which represent the upper and lower boundaries within which potential imaging targets are located.

[0124] For example, a satellite may slew between an upper edge 601 and a lower edge 603, to capture images various imaging targets 604 between the upper edge 601 and the lower edge 603, creating an optimized slew track 602 as it orbits the Earth. For instance, the optimized slew track 602 may indicate an updated trajectory (e.g., output 308) based on output 307 generated by the phenomenon detection model 302 of the satellite system 100. In some examples, the optimized slew track 602 may include an optimized slew trajectory that passes over imaging targets 604A, 604B which include one or more phenomena or include one or more phenomenon targets above the one or more phenomena acquisition parameters. In some examples, the optimized slew track 602 may include the satellite’s pose or orientation (e g., angle, position, etc.) or movement in reference to its orbit track 600. As depicted in FIG. 6, a satellite may navigate along the optimized slew track 602 above or below the orbit track.

[0125] In some examples, the optimized slew track 602 may be determined in consideration of one or more phenomena at various imaging targets 604. For instance, the optimized slew track 602 may capture imaging target 604C based on its percentage of phenomena coverages. For instance, imaging target 604C may be assigned to the satellite 200 traveling along the orbit track 600 as a result of the proximity of the imaging target 604C relative to the orbit track 600. In some examples, imaging target 604C may be assigned to the satellite traveling along the orbit track 600 as a result of the imaging target 604C being positioned within a field of view of the surveillance image sensors 203. For instance, the surveillance image sensor203 may include a fish-eye field of view which encompasses the field of view of the payload sensor 204.

[0126] In some examples, the satellite may detect one or more phenomena at the imaging target 604C which exceed the phenomenon acquisition parameter, and capture imaging target 604C based on detecting the presence of phenomena. For instance, imaging target 604C may be associated with a 30% of one or more phenomena acquisition parameter and determine imaging target 604C is 70% obstructed one or more phenomena. Thus, the satellite may capture imaging target 604C based the one or more phenomena above the one or more phenomena acquisition parameter and instead update the optimized slew track 602 to acquire imagery of imaging target 604B instead.

[0127] For further illustration, after completing image acquisition of imagery target 604 A, the current trajectory data (e.g., current trajectory data 306A) may indicate imagery target 604C as the next target scheduled for image acquisition. The phenomenon detection model (e.g. phenomenon detection model 302) may determine that imagery target 306C does not satisfy the one or more phenomena acquisition parameters, and thus the optimized slew track 602 may be updated by the flight computing system (e.g. flight computing system 107) to slew the satellite to be positioned to capture imagery target 604B instead.

[0128] In some examples, the satellite may include a field of regard and may travel along the optimized slew track 602 within the field of regard. A field of regard may include the total area that a sensing system (e.g., surveillance image sensor 203) can perceive. In some examples, the field of regard may be based on the total area visible to the surveillance image sensor 203. In some examples, the field of regard may be greater than the FOV of the surveillance image sensor 203. For instance, the field of regard may be equal to or greater than the fish-eye field of view of the surveillance image sensor 203. In some examples, the field of regard may include off-nadir geographic regions visible by the payload sensor 204 in addition to the total area visible by the surveillance image sensor 203. For instance, the field of regard may include an upper edge 601 and a lower edge 603. In some embodiments, the surveillance image sensor 203 can include 2, 3, 4, 6, 8, or more sensors to create a field of regard of at least 90 degrees, at least 135 degrees, at least 180 degrees, at least 270 degrees, or 360 degrees about an axis. In some embodiments, the system may have such field of view about multiple (e.g., 2, 3) axes, giving a wide field of view of, for example, both the Earth’s surface and / or outer space. In some examples, the surveillanceimage sensor 203 may capture sensor data 304 depicting geographic regions within the upper edge 601 and lower edge 603 of the field of regard. In some examples, the optimized slew track 602 may be positioned within the upper edge 601 and lower edge 603 of the field of regard due to the sensor data 304 captured by the surveillance image sensor 203 including input images 401 which include geographic regions within the upper edge 601 and lower edge 603. In some examples, the surveillance image sensor 203 may be moveable and cause the upper edge 601 and lower edge 603 of the field of regard to change as the surveillance image sensor 203 moves. As the satellite slews within the upper edge 601 and the lower edge, the payload sensor 204 may be positioned to include an off-nadir field of view greater than the fish-eye field of view of the surveillance image sensor 203.

[0129] FIG. 7A depicts a flow diagram of an example method 700 according to example embodiments of the present disclosure. One or more portion(s) of the method 700 may be implemented by a computing system that includes one or more computing devices such as, for example, the computing systems described with reference to FIGS. 1-6. Each respective portion of the method 700 may be performed by any (or any combination) of one or more computing devices. Moreover, one or more portion(s) of the method 700 may be implemented as an algorithm on the hardware components of the device(s) described herein, for example, to control satellites to capture phenomena. FIG. 7 depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein may be adapted, rearranged, expanded, omitted, combined, and / or modified in various ways without deviating from the scope of the present disclosure. FIG. 7 is described with reference to elements / terms described with respect to other systems and figures for example illustrated purposes and is not meant to be limiting. One or more portions of method 700 may be performed additionally, or alternatively, by other systems.

[0130] At (702), the method 700 may include obtaining image data from one or more surveillance image sensors of a satellite. The method can include obtaining image data from one or more sensors of a satellite comprising a surveillance image sensor. The satellite can be traveling along a current trajectory. As described herein, the sensor data 304 may be captured by a surveillance image sensor 203 of the satellite 200. The surveillance image sensor 203 may be affixed, in some embodiments, in a forward-oriented position on a surface of the satellite 200. Insome embodiments, the surveillance image sensor 203 is disposed to view a large field of regard comprising greater than 120 degrees, such as a fish-eye imager. In some examples, the surveillance image sensor 203 may be a camera sensor. The surveillance image sensor 203 may include a VIS camera, a MWIR camera, and / or any type of camera that can detect phenomena. In some examples, the sensor data 304 including the image data (e.g., input image 401) may be used as input into a machine-learned phenomenon detection model 302.

[0131] At (704), the method 700 may include identifying a phenomenon within at least a portion of a field of view of the one or more surveillance image sensors. Identifying the phenomenon can include accessing metadata associated with one or more environmental conditions. For instance, metadata 305 may include additional data which improves the accuracy of sensor data 304 or confidence level of the phenomenon detection model 302. For example, metadata 305 may include environmental conditions such as map data of historical phenomenon location, monthly snow maps, land cover classifications, camera temperatures, sun angle, time of day, or altitude. Metadata 305 may include any data which may be used to increase the accuracy of sensor data 304 or improve the confidence level of the phenomenon detection model 302. By way of example, metadata 305 including a map of historical phenomenon location may indicate a consistent pattern of one or more phenomena over a geographic region. In some examples, the metadata 305 including the historical phenomenon location may improve a confidence level determination of the phenomenon detection model 302 that a red, orange, yellow, and / or white phenomenon-like formation depicted in the sensor data 304 includes one or more phenomena. In some examples, the metadata 305 may be used as input into the phenomenon detection model 302.

[0132] At (706), the method 700 may include generating a command to capture an image with a payload image sensor having a field of view smaller than the field of view of the one or more surveillance image sensors. The command may be sent to one or more payload sensors (e.g., the payload sensor 204). The one or more payload sensors may be disposed on the satellite or on a second satellite. Accordingly, the command may be transmitted to a second satellite. The command may be routed via a ground station.

[0133] In some embodiments, the method 700 includes determining, using a model and based at least in part on the image data and the metadata, the imaging target and one or more phenomena associated with the imaging target. For instance, the surveillance image sensor 203of the satellite may capture sensor data 304 including an input image 401 indicative of geographic regions within a field of regard of the surveillance image sensor 203. In some examples, sensor data 304 including an input image 401 may be fused with the metadata 305 to compensate for any environmental factors and increase the accuracy of the input image 401. In some examples, a phenomenon detection model 302 may perform candidate pixel determination 402 techniques to identify one or more candidate pixels. The system can determine a candidate pixel region 412 distinct from a control pixel region 408. Determining the candidate pixel region 412 can include segmenting the input image 401 fused with the metadata 305 based on objects (e.g., phenomena) or imaging targets depicted in the input image 401 fused with the metadata 305. In some examples, the phenomenon detection model 302 may generate an up-sampling to up-sample the image segment.

[0134] The phenomenon detection model 302 may determine the up-sampled image segment depicts one or more phenomena and may generate labels indicative of phenomenon characteristics. For instance, the phenomenon detection model 302 may generate a label indicating the object depicted in the up-sampled image segment is a phenomenon and generate a label indicating the level of one or more phenomena.

[0135] By way of example, the phenomenon detection model 302 may analyze the up-sampled image segments and generate a bounding shape (e.g., a candidate pixel region boundary 416) to encapsulate phenomenon-like objects depicted in the image frame. In some examples, the phenomenon detection model 302 may identify a phenomenon depicted in the bounding shape and generate a phenomenon label. In some examples, the phenomenon detection model 302 may determine a level of phenomenon coverage depicted in an up-sampled image frame and generate a label indicating the level of phenomenon coverage.

[0136] In some examples, the phenomena-coverage label may be associated with a phenomenon label. In other examples, the phenomena-coverage label may be nested within a phenomenon label. Determining a level of phenomena may include determining the percentage of pixels associated with the detected phenomenon relative to the bounding shape within the up-sampled image segment. In some examples, the phenomenon detection model 302 may determine based on metadata 305 the level of one or more phenomena within the up-sampled image frame. For instance, Earth surface angle metadata may indicate that the detected phenomena are not dense enough to obscure an image acquisition. In some examples, the level ofone or more phenomena may be determined based on metadata indicating the density of phenomena depicted in the up-sampled image segment.

[0137] In some examples, the phenomenon detection model 302 may detect phenomena within the up-sampled image segment and generate a phenomena-coverage label to indicate the level of one or more phenomena. The phenomena-coverage label may include an integer value, percentage value, or any value which indicates a consistent measure of one or more phenomena.

[0138] In some embodiments, the method 700 may include determining a comparison between the one or more phenomena associated with the imaging target and a threshold level of one or more phenomena. For instance, the current trajectory data 306A may include data associated with a phenomenon threshold or tolerance level for respective imaging targets. For example, the current trajectory data 306A, may indicate imaging targets scheduled for image acquisition. In some examples, the imaging targets scheduled for image acquisition may indicate additional acquisition parameters. For instance, the imaging targets may indicate a phenomenon threshold or tolerance level to satisfy the acquisition request. The phenomenon threshold or tolerance level may include an integer value, percentage value, or any value which indicates a consistent measure of one or more phenomena. In some examples, the phenomenon threshold or tolerance level may include an upper limit threshold, lower limit threshold or an exact threshold or tolerance level. In some examples, satisfying the threshold or tolerance level may include a phenomena-coverage above the phenomenon threshold, below the phenomenon threshold or exactly matching the phenomenon threshold.

[0139] The phenomenon detection model 302 may compare the phenomena-coverage label of the up-sampled image segment to the phenomenon threshold or tolerance level for the requested imagery of the geographic region and determine imaging targets which satisfy the phenomenon threshold acquisition parameter.

[0140] In some embodiments, the method 700 may include determining an updated trajectory for the satellite based on the current trajectory and the comparison. For instance, the phenomenon detection model 302 may determine one or more imaging targets which satisfy the phenomenon threshold acquisition parameter based on the comparison between the phenomena-coverage label of the up-sampled image segment to the phenomenon threshold or tolerance level for the requested imagery of the geographic region.

[0141] In some examples, the phenomenon detection model 302 may output 307 indicative of an updated trajectory which includes the imaging targets that satisfy the phenomenon threshold acquisition parameters. In some examples, the phenomenon detection model 302 may transmit output 307 to a flight controller 301 of the flight computing system 107. In some examples the flight controllers 301 may be configured to receive the output 307 and generate an updated trajectory based on the output 307.

[0142] In some embodiments, the method 700 may include generating one or more command instructions to control a motion of the satellite based at least in part on the updated trajectory. For instance, the flight controllers 107 of the flight computing system 107 may determine the current trajectory does not include a slew trajectory which passes over the imaging targets or a possible off-nadir position of the payload sensor 204 indicated in the output 307. For instance, the payload sensor 204 may acquire imagery off-nadir (e.g., not directly below the satellite 200). In some examples, the off-nadir imagery may be captured by the satellite as it travels along the slew trajectory. In some examples, the flight computing system 107 may determine that an off-nadir image is not possible based on the current trajectory (e.g., position, orientation, etc.) of the satellite. In other examples, the flight computing system 107 may determine an off-nadir image would be distorted based on the current trajectory of the satellite. The flight computing system 107 may generate an updated trajectory which includes waypoints, etc. that pass over the imaging targets indicated in the output 307. For instance, the flight controllers 301 may generate output 308 to control a motion of the satellite system. In some examples the output 308 may include one or more command instructions to control a motion of the satellite based on the updated trajectory.

[0143] While FIG. 7A describes the determination of imaging targets for a single satellite system 100, the present disclosure is not limited to such an embodiment. The imaging targets may be communicated to other satellites in a constellation. For instance, the model may determine an imaging target that the satellite will not pass over and communicate the imaging target to another satellite which has a higher probability of passing over the imaging target. As such the present disclosure may be implemented by a plurality (e.g., constellation) of imaging satellites.

[0144] By way of example, the phenomenon detection model 302 may determine based on the current schedule of pending image acquisitions and associated image acquisitionparameters that acquiring the requested imagery for an imaging target is unlikely. For instance, the phenomenon detection model 302 may prioritize imaging targets based on commercial factors such that a threshold number of orbits will be required to acquire the requested imagery for the imaging target. The threshold number of orbits may include 2 orbits, 3 orbits, or any number of orbits. In some examples, the threshold number of orbits may be based on a time acquisition parameter. For instance, the time acquisition parameter may indicate that imagery for the imaging target must be acquired prior to a priority status associated with the imaging target. The satellite system 100 may communicate via the communication system 106 data indicating the image acquisition command cannot be satisfied by the respective satellite 200. In some examples, the satellite system 100 may communicate with a ground station. In other examples, the satellite system 100 may communicate with other satellites 200 in a constellation. For instance, a second satellite 200 may include an optimal orbit track 600, or trajectory (e g., slew trajectory) which is capable of acquiring the requested imagery. In some examples, multiple satellites 200 may be spaced along the same or similar orbit track 600.

[0145] In some examples, the satellite 200 may be occupied with another task and is unable to satisfy the image acquisition command. For instance, the satellite 200 may be tasked with downlinking acquired imagery and is unable to generate an updated trajectory to downlink acquired imagery and satisfy the image acquisition command. In some examples, the satellite 200 may communicate with a ground station (e.g., during a downlink) or directly with other satellites 200 in the constellation to indicate the inability to satisfy the image acquisition command.

[0146] In some examples, the satellites in the constellation may include different payload capabilities. For instance, the satellite 200 may include a payload sensor 204 with a deep focus lens. In some examples, the satellite 200 may require a payload sensor with a shallow focus lens due to a trajectory which causes an off-nadir imaging position, less-closely oriented position relative to the imaging target, etc. In some examples, another satellite within the satellite constellation (or a different satellite constellation) may include a shallow focus lens necessary for satisfying the image acquisition command. Similarly, other satellites may include different sensor characteristics which may be preferred or which may be more capable for collection of data from a given geographic area. For example, the satellite 200 may include a payload sensor 204 which is an optical or multi-spectral, but which does not possess other payload sensors.Another satellite which is under the control of the same satellite operator may include a different payload sensor, for example including, hyperspectral, infrared, NIR (near-infrared), SWIR (short-wave infrared), MWIR (mid-wave infrared), radar (including synthetic-aperture radar (SAR)), etc. Similarly, other satellites may include different resolution and / or capture size (e.g., swath width) characteristics. In some examples, the satellite 200 may communicate with another satellite which includes the capabilities required or desired to acquire the requested imagery.

[0147] A second satellite in a satellite constellation may receive the image acquisition command from a first satellite and generate an updated trajectory to pass over the associated imaging target. In some examples, the second satellite may align a payload sensor with the imaging target and acquire the rested imagery. In some examples, the second satellite may pass over a geographic region near the imaging target and capture nadir or off-nadir imagery of the imaging target.

[0148] In some examples, satellite 200 may have low spatial resolution. Given this, satellite 200 may be of relatively low cost and small size. The satellite 200 may send an image acquisition command to a second satellite which has higher resolution. In this way, the satellite 200 can act as a scout for the higher resolution satellite, such that the more valuable payload sensor is used more efficiently based on received image acquisition commands from the (first) lower cost, satellite 200.

[0149] In some examples, satellites having different payload sensor 204 capabilities are impacted differently by phenomenon coverage or intensity. For example, the satellite 200 may have a payload sensor 204 which is impacted by phenomena (e.g., optical sensor), and thus a given imaging target may not be suitable for imagery capture given one or more phenomena, as determined by the phenomenon detection model 302. However, a second satellite may include a different payload sensor which is not impacted, or is less impacted, by phenomenon intensity (e.g. synthetic-aperture radar (SAR)). In such circumstances, the satellite 200 may send an acquisition command to the second satellite to acquire data (using SAR, for example) from the phenomenon-obstructed area. That may, for example, provide an alternative to the satellite operator to collect data of the imagery target, as an alternative to optical data given the one or more phenomena, and could provide an alternative imagery collection path which could utilize analytic techniques to synthetically process optical imagery to enhance phenomenon detection.

[0150] FIG. 7B depicts a flow diagram of an example method 701 according to example embodiments of the present disclosure. One or more portion(s) of the method 701 may be implemented by a computing system that includes one or more computing devices such as, for example, the computing systems described with reference to FIGS. 1-6. Each respective portion of the method 701 may be performed by any (or any combination) of one or more computing devices. Moreover, one or more portion(s) of the method 701 may be implemented as an algorithm on the hardware components of the device(s) described herein, for example, to control satellites to capture phenomena. FIG. 7B depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein may be adapted, rearranged, expanded, omitted, combined, and / or modified in various ways without deviating from the scope of the present disclosure. FIG. 7B is described with reference to elements / terms described with respect to other systems and figures, for example illustrated purposes and is not meant to be limiting. One or more portions of method 701 may be performed additionally, or alternatively, by other systems.

[0151] At (712), the method 701 may include obtaining image data from surveillance image sensors of a satellite. For instance, the surveillance image sensor 203 may include a look ahead angle 202 from nadir. At (714), the method 701 may include identify a phenomenon within at least a portion of a field of view of the one or more surveillance image sensors.Identifying the phenomenon may include determining, using a model and based on the image data, an imaging target and one or more phenomena associated with the imaging target. At (714), the method 701 may include other steps described herein. For instance, the phenomenon detection model 302 may obtain an input image 401. Input image 401 may be sensor data 304 (e g., image data) captured by the surveillance image sensor 203 of the satellite 200. In some examples, the input image 401 may include an image frame fused with metadata 305. The input image 401 may include a plurality of imaging targets and the input image 401 may include respective imaging targets within the input image 401.

[0152] In some embodiments, the method 701 includes analyzing, using a model an image frame of the image data. For instance, the phenomenon detection model 302 may analyze the sensor data 304 (e.g., input image frame 401) fused with the metadata 305 and project a bounding shape on the image frame. In some examples, the bounding shape may encapsulategeographic regions associated with imaging targets depicted in the image frame. Tn other examples, the bounding shape may encapsulate objects (e.g., phenomena, landmasses, water, ice, etc.) depicted in the image frame. In some examples, the phenomenon detection model 302 may process the image frames to determine one or more phenomena associated with the imaging targets scheduled for image acquisition.

[0153] The method 701 may include generating, using the model, a plurality of image segments, wherein the plurality of image segments is associated with one or more phenomena depicted in the image frame. The phenomenon detection model 302 may perform candidate pixel determination 402 to segment the input image 401 based on detecting one or more imaging targets. In some examples, the phenomenon detection model 302 may perform candidate pixel determination 402 to segment the input image 401 based on detecting one or more objects.

[0154] In some embodiments, the method 701 can include determining a set of coordinates corresponding to or indicative of a boundary of the field of view of the surveillance image sensor. The method 701 can include generating a grid within the boundary of the field of view of the surveillance image sensor. The grid can include a plurality of cells. The system can identify the phenomenon within the field of view of the surveillance image sensor by determining a location of the phenomenon within one or more of the cells of the grid. The system may use these cells to up-sample the grid to generate a second grid having a plurality of up-sampled cells. In some embodiments, the system can determine a location of the phenomenon within one or more of the up-sampled cells.

[0155] The method 701 may include generating, using the model, a plurality of up-sampled image segments, wherein the plurality of up-sampled image segments are indicative of one or more phenomenon characteristics. For instance, the phenomenon detection model 302 may perform candidate pixel determination 402 to segment the input image 401 based on detecting one or more imaging targets and generate an up-sampling 403 to up-sample the input image 401 (e.g., image segments). In some examples, the up-sampling 403 may increase the clarity or sharpness of at least a portion of the input image 401.

[0156] The phenomenon detection model 302 may analyze the up-sampled image segments and determine one or more characteristics indicative of one or more phenomena. For instance, the phenomenon detection model 302 may generate characteristics data (e.g., labels) that correspond to the characteristics of the bounding shape. Labels may include theclassification of objects (e.g., land, water, etc.), the type of objects (e.g., phenomena, snow, ocean foam, etc.), density, etc. In some examples, the characteristics data (e.g., labels) may indicate the presence of phenomena in the up-sampled image segment. In some examples, the characteristics data may indicate the absence of phenomena in the up-sampled image segment. In some examples, the up-sampling 403 may reduce the characteristics data needed to identify one or more phenomena or the absence of phenomena.

[0157] In some embodiments, the method 701 may include determining, using the model, the one or more phenomena associated with respective up-sampled image segments of the plurality of up-sampled image segments. For instance, the phenomenon detection model 302 may generate a phenomena-coverage label. In some examples, the phenomena-coverage label may be associated with a phenomenon label. In other examples, the phenomena-coverage label may be nested within a phenomenon label. The phenomena-coverage may be a percentage, ratio, or any measure of one or more phenomena relative to the up-sampled image frame (e.g., geographic target). By way of example, the phenomenon detection model 302 may analyze an up-sampled image segment and detect the presence of phenomena. In some examples, the phenomenon detection model 302 may determine a level of phenomena within the up-sampled image frame. Determining a level of phenomena may include determining the percentage of pixels associated with the detected phenomenon relative to the bounding shape within the up-sampled image segment. In some examples, the phenomenon detection model 302 may determine based on metadata 305 the level of one or more phenomena within the up-sampled image frame. For instance, Earth surface angle metadata may indicate that the detected phenomena are not dense enough to obscure an image acquisition. In some examples, the level of one or more phenomena may be determined based on metadata 305 indicating the density of phenomena depicted in the up-sampled image segment.

[0158] In some examples, the phenomenon detection model 302 may detect phenomena within the up-sampled image segment and generate a phenomena-coverage label to indicate the level of one or more phenomena. The phenomena-coverage label may include an integer value, percentage value, ratio, or any value which indicates a consistent measure of one or more phenomena.

[0159] At (716), the method may include transmitting, to a second satellite comprising a payload image sensor, a command to capture an image with the payload image sensor. For example, the second satellite may include one or more of the features described herein.

[0160] While FIG. 7B describes the determination of one or more phenomena depicted in a single image frame, the present disclosure is not limited to such an embodiment. The determination of one or more phenomena may be determined across a plurality of image frames. For instance, the model may determine one or more phenomena by analyzing, segmenting, and up-sampling a plurality of image frames captured over a period of time. As such the present disclosure may be implanted over a period of time as the satellite travels along its orbit.

[0161] FIG. 8 depicts an example computing system 800 that may be used to implement the methods and systems according to example aspects of the present disclosure. The system 800 may include computing system 805 (e.g., ground station) and satellite 855 (e.g., satellite system 100), which may communicate with one another using transmission signals 810 (e.g., radio frequency transmissions). The system 800 may be implemented using a client-server architecture and / or other suitable architectures.

[0162] The computing system 805 may include one or more computing device(s) 815. Computing device(s) 815 may include one or more processor(s) 820 and one or more memory device(s) 825. Computing device(s) 815 may also include a communication interface 840 used to communicate with satellite 855 and / or another computing system / device. Communication interface 840 may include any suitable components for communicating with satellite 855 and / or another system / device, including for example, transmitters, receivers, ports, controllers, antennas, or other suitable components.

[0163] Processor(s) 820 may include any suitable processing device, such as a microprocessor, microcontroller, integrated circuit, logic device, or other suitable processing device. Memory device(s) 825 may include one or more computer-readable media, including, but not limited to, non-transitory computer-readable media, RAM, ROM, hard drives, flash drives, or other memory devices. Memory device(s) 825 may store information accessible by processor(s) 820, including computer-readable instructions 830 that may be executed by processor(s) 820. Instructions 830 may be any set of instructions that when executed by processor(s) 820, cause one or more processor(s) 820 to perform operations. For instance, execution of instructions 830 may cause processor(s) 820 to perform any of the operations and / orfunctions for which computing device(s) 915 and / or computing system 805 are configured (e g., such as the functions of a satellite ground station, GEO hub, etc.). In some implementations, execution of instructions 830 may cause processor(s) 820 to perform, at least a portion of, methods 700 according to example embodiments of the present disclosure.

[0164] As shown in FIG. 8, memory device(s) 825 may also store data 835 that may be retrieved, manipulated, created, or stored by processor(s) 820. Data 835 may include, for instance, any other data and / or information described herein. Data 835 may be stored in one or more database(s). The one or more database(s) may be connected to computing device(s) 815 by a high bandwidth LAN or WAN, or may also be connected to computing device(s) 815 through various other suitable networks. The one or more databases may be split up so that they are located in multiple locales.

[0165] The computing system 805 may include a model trainer 845 that trains the machine-learned models 885 stored at the satellite 855 using various training or learning techniques. For example, the machine-learned models 885 may be trained using a loss function. By way of example, fortraining a phenomenon detection model, the model trainer 845 may use a loss function. For example, a loss function may be backpropagated through the machine-learned models 885 to update one or more parameters of the machine-learned models 885 (e.g., based on a gradient of the loss function). Various loss functions may be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters over a number of training iterations.

[0166] The model trainer 845 may train the machine-learned models 885 (e.g., phenomenon detection model) in an unsupervised fashion. As such, the machine-learned models 885 may be effectively trained using unlabeled data for particular applications or problem domains, which improves performance and adaptability of the machine-learned models 885.

[0167] The computing system 805 may modify parameters of the machine-learned models 885 (e.g., the phenomenon detection model 302) based on the loss function such that the machine-learned models 885 may be effectively trained for specific applications in an unsupervised manner without labeled data.

[0168] The model trainer 845 may utilize training techniques, such as backwards propagation of errors. For example, a loss function may be backpropagated through a model toupdate one or more parameters of the models (e.g., based on a gradient of the loss function). Various loss functions may be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters over a number of training iterations.

[0169] In an embodiment, performing backwards propagation of errors may include performing truncated backpropagation through time. The model trainer 845 may perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of a model being trained. In particular, the model trainer 845 may train the machine-learned models 885 based on a set of training data 850.

[0170] The training data 850 may include unlabeled training data for training in an unsupervised fashion. In an example, the training data 850 may include unlabeled sets of data indicative of phenomenon formations, snow / ice covered land, sea foam, etc. The training data 850 may be specific to satellite or constellation of satellites to help focus the machine-learned models 885 on a particular orbital pattern.

[0171] In an embodiment, training examples may be provided by the satellite 855 (e.g., satellite system 100). Thus, in such implementations, a model 885 provided to the satellite 855 may be trained by the computing system 805 in a manner to personalize the model 885.

[0172] The model trainer 845 may include computer logic utilized to provide desired functionality. The model trainer 845 may be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in an embodiment, the model trainer 845 may include program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer 845 may include one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM, hard disk, or optical or magnetic media.

[0173] Computing system 805 may exchange data with satellite 855 using signals 810. Although one satellite 855 is illustrated in FIG. 8, any number of satellites may be configured to communicate with the computing system 805. In some implementations, satellite 855 may be associated with any suitable type of satellite system, including satellites, mini-satellites, microsatellites, nano-satellites, etc. Satellite 855 may correspond to any of the satellites described herein (e.g., satellite system 100.).

[0174] Satellite 855 may include computing device(s) 860, which may include one or more processor(s) 865 and one or more memory device(s) 870. Processor(s) 865 may include one or more central processing units (CPUs), graphical processing units (GPUs), and / or other types of processors. Memory device(s) 870 may include one or more computer-readable media and may store information accessible by processor(s) 865, including instructions 875 that may be executed by processor(s) 865. For instance, memory device(s) 870 may store instructions 875 for implementing a command receive and image collect for capture image data; storing image data, commands, tracks, etc.; transmitting the image data to a remote computing device (e.g., computing system 805). In some implementations, execution of instructions 875 may cause processor(s) 865 to perform any of the operations and / or functions for which satellite 100 is configured. In some implementations, execution of instructions 875 may cause processor(s) 865 to perform, at least a portion of, method 700.

[0175] Memory device(s) 870 may also store data 880 that may be retrieved, manipulated, created, or stored by processor(s) 865. Data 880 may include, for instance, image acquisition commands, tracks, sequences, position data, data associated with the satellite, image data, and / or any other data and / or information described herein. Data 880 may be stored in one or more database(s). The one or more database(s) may be connected to computing device(s) 860 by a high bandwidth LAN or WAN, or may also be connected to computing device(s) 860 through various other suitable networks. The one or more database(s) may be split up so that they are located in multiple locales.

[0176] In an embodiment, the satellite 855 may store or include one or more machine-learned models 885. For example, the machine-learned models 885 may be or may otherwise include various machine-learned models. In an embodiment, the machine-learned models 885 may include neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine-learned models may leverage an attention mechanism such as self-attention. For example, some example machine-learned models may include multi-headed self-attention models (e.g., transformer models).

[0177] In an embodiment, the one or more machine-learned models 885 may be received from the computing system 805 via one or more signals 810, stored in the satellite 855 (e.g., memory 870), and then used or otherwise implemented by the processors 865. In an embodiment, the satellite 855 may implement multiple parallel instances of a single model.

[0178] Additionally, or alternatively, one or more machine-learned models 885 may be included in or otherwise stored and implemented by the computing system 805 that communicates with the satellite 855 according to a client-server relationship. For example, the machine-learned models 885 may be implemented by the computing system 805 as a portion of GEO communication infrastructure. Thus, one or more models 885 may be stored and implemented at the satellite 855 and / or one or more models 885 may be stored and implemented at the computing system 805.

[0179] Satellite 855 may also include a communication interface 890 used to communicate with one or more remote computing device(s) (e.g., computing system 805, geostationary satellite(s), etc.) using signals 810. Communication interface 890 may include any suitable components for interfacing with one or more remote computing device(s), including for example, transmitters, receivers, ports, controllers, antennas, or other suitable components.

[0180] In some implementations, one or more aspect(s) of communication among the components of system 800 may involve communication through a network. In such implementations, the network may be any type of communications network, such as a local area network (e.g. intranet), wide area network (e.g. Internet), cellular network, or some combination thereof. The network may also include a direct connection, for instance, between one or more of the components. In general, communication through the network may be carried via a network interface using any type of wired and / or wireless connection, using a variety of communication protocols (e.g. TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g. HTML, XML), and / or protection schemes (e.g. VPN, secure HTTP, SSL).

[0181] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. One of ordinary skill in the art will recognize that the inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, server processes discussed herein may be implemented using a single server or multiple serversworking in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0182] Furthermore, computing tasks discussed herein as being performed at a server may instead be performed at a user device. Likewise, computing tasks discussed herein as being performed at the user device may instead be performed at the server.

[0183] While the present subject matter has been described in detail with respect to specific example embodiments and methods thereof, it will be appreciated that those skilled in the art, upon attaining an understanding of the foregoing may readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the scope of the present disclosure is by way of example rather than by way of limitation, and the subject disclosure does not preclude inclusion of such modifications, variations and / or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art.Example Embodiments

[0184] The following provides a non-limiting exemplary set of example embodiments described herein.

[0185] In a 1st Example, a computer system comprising: a computer readable medium storing computer executable instructions; and one or more hardware processors in communication with the computer readable medium, and configured to execute the computer executable instructions in order to: obtain image data from one or more surveillance image sensors of a satellite; identify, based on the image data, a phenomenon within at least a portion of a field of view of the one or more surveillance image sensors; and generate, based on identifying the phenomenon within the at least the portion of the field of view of the one or more surveillance image sensors, a command to capture an image with a payload image sensor having a field of view smaller than the field of view of the one or more surveillance image sensors.

[0186] In a 2nd Example, the computer system of Example 1, wherein the satellite comprises the payload image sensor, and wherein capturing the image with the payload image sensor comprises modifying an orientation of the satellite.

[0187] In a 3rd Example, the computer system of Example 2, wherein modifying the orientation of the payload image sensor comprises causing the payload image sensor to be oriented toward the identified phenomenon.

[0188] In a 4th Example, the computer system of any of Examples 1-3, wherein the one or more hardware processors are further configured to execute the computer executable instructions in order to: determine a time window in which the payload image sensor is capable of imaging the phenomenon, wherein generating the command is further based on the determined time window.

[0189] In a 5th Example, the computer system of any of Examples 1-4, wherein the one or more surveillance image sensors are configured to obtain a field of view of at least 90 degrees and to obtain images in a wavelength range of between 3 microns and 8 microns.

[0190] In a 6th Example, the computer system of any of Examples 1-5, wherein the one or more surveillance image sensors comprises a plurality of surveillance image sensors configured to obtain a total field of view of at least 180 degrees.

[0191] In a 7th Example, the computer system of any of Examples 1-6, wherein identifying the phenomenon comprises: comparing, based on the image data, an intensity of a plurality of candidate pixels to an intensity of a plurality of control pixels; and determining that the comparison of the intensity of the plurality of candidate pixels to the intensity of the plurality of control pixels satisfies a phenomenological threshold.

[0192] In an 8th Example, the computer system of Example 7, wherein comparing the intensity of the plurality of candidate pixels to the intensity of the plurality of control pixels comprises: extracting, using a model, a plurality of features associated with the plurality of candidate pixels.

[0193] In a 9th Example, the computer system of any of Examples 1-8, wherein the one or more hardware processors are further configured to execute the computer executable instructions in order to: determine a set of coordinates indicative of a boundary of the field of view of the one or more surveillance image sensors.

[0194] In a 10th Example, the computer system of Example 9, wherein the one or more hardware processors are further configured to execute the computer executable instructions in order to: generate a grid within the boundary of the field of view of the one or more surveillance image sensors, the grid comprising a plurality of cells, wherein identifying the phenomenonwithin at least a portion of the field of view of the one or more surveillance image sensors comprises determining a location of the phenomenon within one or more of the plurality of cells.

[0195] In an 11th Example, the computer system of Example 10, wherein the one or more hardware processors are further configured to execute the computer executable instructions in order to: up-sample, based on determining the location of the phenomenon within the one or more cells, the grid to generate a second grid having a plurality of up-sampled cells; and determine a location of the phenomenon within one or more of the plurality of up-sampled cells.

[0196] In a 12th Example, the computer system of any of Examples 10-11, wherein the one or more hardware processors are further configured to execute the computer executable instructions in order to: determine a pixel intensity of one or more pixels associated with the one or more of the plurality of cells; and determine that the pixel intensity of the one or more pixels exceeds a threshold.

[0197] In a 13th Example, the computer system of any of Examples 1-12, wherein the one or more surveillance image sensors comprise a microwave radiometer or bolometer configured to sense at least one of radio frequency (RF) waves, microwaves, or terahertz frequency waves.

[0198] In a 14th Example, the computer system of any of Examples 1-13, wherein identifying the phenomenon within the at least the portion of the field of view of the one or more surveillance image sensors comprises: obtaining second image data from the one or more surveillance image sensors; comparing the image data to the second image data; and determining, based on comparing the image data to the second image data, that the phenomenon is not associated with an artifact.

[0199] In a 15th Example, a computer-implemented method comprising: obtaining, using a payload image sensor of a satellite, image data from one or more surveillance image sensors of a satellite, wherein the one or more surveillance image sensors are configured to obtain a field of view of at least 90 degrees; identifying, based on the image data, a phenomenon within at least a portion of the field of view of the one or more surveillance image sensors; and generating, based on identifying the phenomenon within the at least the portion of the field of view of the one or more surveillance image sensors, a command configured to modify an orientation of the payload image sensor.

[0200] In a 16th Example, the method of Example 1 , further comprising: determining a set of coordinates indicative of a boundary of the field of view of the one or more surveillance image sensors.

[0201] In a 17th Example, the method of Example 16, further comprising: generating a grid within the boundary of the field of view of the one or more surveillance image sensors, the grid comprising a plurality of cells, wherein identifying the phenomenon within at least a portion of the field of view of the one or more surveillance image sensors comprises determining a location of the phenomenon within one or more of the plurality of cells.

[0202] In an 18th Example, a computer system comprising: a computer readable medium storing computer executable instructions; and one or more hardware processors in communication with the computer readable medium, and configured to execute the computer executable instructions in order to: obtain image data from one or more surveillance image sensors of a satellite; identify, based on the image data, a phenomenon within at least a portion of a field of view of the one or more surveillance image sensors; and transmit, to a second satellite comprising a payload image sensor, a command to capture an image with the payload image sensor.

[0203] In a 19th Example, a computer-implemented method comprising: obtaining image data from one or more surveillance image sensors of a satellite; identifying, based on the image data, a phenomenon within at least a portion of a field of view of the one or more surveillance image sensors; and transmitting, to a second satellite comprising a payload image sensor, a command to capture an image with the payload image sensor.

[0204] In a 20th Example, the method of Example 19, further comprising: determining a time window in which the payload image sensor is capable of imaging the phenomenon.

[0205] In a 21st Example, the method of any of Examples 19-20, further comprising: determining a set of coordinates indicative of a boundary of the field of view of the one or more surveillance image sensors.

Claims

1. WHAT IS CLAIMED IS:

1. A computer system comprising:a computer readable medium storing computer executable instructions; and one or more hardware processors in communication with the computer readable medium, and configured to execute the computer executable instructions in order to:obtain image data from one or more surveillance image sensors of a satellite;identify, based on the image data, a phenomenon within at least a portion of a field of view of the one or more surveillance image sensors; and generate, based on identifying the phenomenon within the at least the portion of the field of view of the one or more surveillance image sensors, a command to capture an image with a payload image sensor having a field of view smaller than the field of view of the one or more surveillance image sensors.

2. The computer system of claim 1, wherein the satellite comprises the payload image sensor, and wherein capturing the image with the payload image sensor comprises modifying an orientation of the satellite.

3. The computer system of claim 2, wherein modifying the orientation of the payload image sensor comprises causing the payload image sensor to be oriented toward the identified phenomenon.

4. The computer system of claim 1, wherein the one or more hardware processors are further configured to execute the computer executable instructions in order to:determine a time window in which the payload image sensor is capable of imaging the phenomenon, wherein generating the command is further based on the determined time window.

5. The computer system of claim 1, wherein the one or more surveillance image sensors are configured to obtain a field of view of at least 90 degrees and to obtain images in a wavelength range of between 3 microns and 8 microns.

6. The computer system of claim 1, wherein the one or more surveillance image sensors comprises a plurality of surveillance image sensors configured to obtain a total field of view of at least 180 degrees.

7. The computer system of claim 1, wherein identifying the phenomenon comprises:comparing, based on the image data, an intensity of a plurality of candidate pixels to an intensity of a plurality of control pixels; anddetermining that the comparison of the intensity of the plurality of candidate pixels to the intensity of the plurality of control pixels satisfies a phenomenological threshold.

8. The computer system of claim 7, wherein comparing the intensity of the plurality of candidate pixels to the intensity of the plurality of control pixels comprises:extracting, using a model, a plurality of features associated with the plurality of candidate pixels.

9. The computer system of claim 1, wherein the one or more hardware processors are further configured to execute the computer executable instructions in order to:determine a set of coordinates indicative of a boundary of the field of view of the one or more surveillance image sensors.

10. The computer system of claim 9, wherein the one or more hardware processors are further configured to execute the computer executable instructions in order to:generate a grid within the boundary of the field of view of the one or more surveillance image sensors, the grid comprising a plurality of cells, wherein identifying the phenomenon within at least a portion of the field of view of the one or more surveillance image sensors comprises determining a location of the phenomenon within one or more of the plurality of cells.

11. The computer system of claim 10, wherein the one or more hardware processors are further configured to execute the computer executable instructions in order to:up-sample, based on determining the location of the phenomenon within the one or more cells, the grid to generate a second grid having a plurality of up-sampled cells; and determine a location of the phenomenon within one or more of the plurality of up- sampled cells.

12. The computer system of claim 10, wherein the one or more hardware processors are further configured to execute the computer executable instructions in order to:determine a pixel intensity of one or more pixels associated with the one or more of the plurality of cells; anddetermine that the pixel intensity of the one or more pixels exceeds a threshold.

13. The computer system of claim 1, wherein the one or more surveillance image sensors comprise a microwave radiometer or bolometer configured to sense at least one of radio frequency (RF) waves, microwaves, or terahertz frequency waves.

14. The computer system of claim 1, wherein identifying the phenomenon within the at least the portion of the field of view of the one or more surveillance image sensors comprises:obtaining second image data from the one or more surveillance image sensors; comparing the image data to the second image data; anddetermining, based on comparing the image data to the second image data, that the phenomenon is not associated with an artifact.

15. A computer-implemented method comprising:obtaining, using a payload image sensor of a satellite, image data from one or more surveillance image sensors of a satellite, wherein the one or more surveillance image sensors are configured to obtain a field of view of at least 90 degrees;identifying, based on the image data, a phenomenon within at least a portion of the field of view of the one or more surveillance image sensors; andgenerating, based on identifying the phenomenon within the at least the portion of the field of view of the one or more surveillance image sensors, a command configured to modify an orientation of the payload image sensor.

16. The method of claim 15, further comprising:determining a set of coordinates indicative of a boundary of the field of view of the one or more surveillance image sensors.

17. The method of claim 16, further comprising:generating a grid within the boundary of the field of view of the one or more surveillance image sensors, the grid comprising a plurality of cells, wherein identifying the phenomenon within at least a portion of the field of view of the one or more surveillance image sensors comprises determining a location of the phenomenon within one or more of the plurality of cells.

18. A computer-implemented method comprising:obtaining image data from one or more surveillance image sensors of a satellite;identifying, based on the image data, a phenomenon within at least a portion of a field of view of the one or more surveillance image sensors; and transmitting, to a second satellite comprising a payload image sensor, a command to capture an image with the payload image sensor.

19. The method of claim 18, further comprising:determining a time window in which the payload image sensor is capable of imaging the phenomenon.

20. The method of claim 18, further comprising:determining a set of coordinates indicative of a boundary of the field of view of the one or more surveillance image sensors.

Images