45 results about "Outlier removal" patented technology

The invention discloses a spectral gas concentration inversion method based on a CNN-Transform fusion model, and belongs to the technical field of trace gas detection and spectral signal processing. The method comprises the following steps: acquiring second harmonic spectrum data of target gas through a TDLAS (tunable diode laser absorption spectroscopy) system; carrying out abnormal value elimination and smooth filtering processing on the acquired signal, and constructing a standard input vector; a depth model fusing a one-dimensional convolutional neural network (CNN) and a Transform architecture is designed, the CNN is used for extracting local spectrogram features, and the Transform is used for modeling a global dependency relationship; and inputting the training set and the verification set into the model for training optimization, and finally realizing high-precision inversion of the sample gas concentration of the test set. Experimental results show that the method is superior to an existing model in the aspects of fitting precision, robustness and error control, and has good physical consistency and engineering popularization value.

Aspects of the invention include training an optimal interpretable decision tree for regression using mixed-integer linear programming techniques. A non-limiting example computer-implemented method includes receiving, using a processor, input data that includes time-series data. The method further includes training, using a binary mixed-integer linear program of the processor, an ODT for regression based on the input data. During the training process one or more outliers are filtered out by a linear loss model that minimizes training loss and outlier loss.

The invention provides a refrigerator fault prediction method based on a gradient boosting tree. The method comprises the steps of data acquisition, missing value processing and abnormal value removal preprocessing. Carrying out feature coding, temperature difference feature calculation and high-dimensional transformation feature extraction to obtain data features; performing time window flattening operation on the data features, performing resistance feature selection and recursive feature elimination, and screening out an optimal feature subset; training an xgboost gradient boosting tree model by using the optimal feature subset, and outputting a fault prediction model when the accuracy rate accords with a preset value; and finally, whether the refrigerator breaks down or not is judged through the fault prediction model. According to the application, comprehensive monitoring and fault prediction of the running state of the refrigerator are realized. The prediction accuracy and generalization ability of the model are improved by improving the data quality and enhancing the model performance; real-time prediction is achieved, and potential faults of the refrigerator can be found in time; and the accuracy and adaptability of prediction are improved by dynamically adjusting the fault threshold.

The invention provides a method for predicting a memory effect of a natural gas hydrate, belongs to the technical field of intelligent modeling and prediction of the natural gas hydrate, and aims to realize accurate modeling and intelligent identification of a memory effect behavior in a hydrate formation and dissociation process. The method comprises the following steps: S1, collecting experimental data, including a plurality of variable characteristics influencing the memory effect, such as synthesis temperature, dissociation pressure, synthesis pressure, synthesis-decomposition cycle index, decomposition temperature and the like, and constructing a training data set by taking nucleation time as a dependent variable; s2, preprocessing the experimental data, completing missing value filling, abnormal value elimination and standardization processing, and obtaining a standardized sample; s3, constructing a prediction model based on support vector regression, random forest, XGBoost, polynomial regression and other algorithms, and improving prediction performance by adjusting model hyper-parameters; and S4, a graphic visual interface is constructed based on Python and PyQt5, a user can input experimental conditions and select a model, and a system automatically predicts a memory effect and judges whether a nucleation promoting behavior exists according to the memory effect. The method integrates key modules such as feature engineering, active learning, hyper-parameter optimization and visual interaction, has high precision, high adaptability and good expansibility, and is suitable for various thermodynamic application scenes such as natural gas hydrate phase change behavior prediction, energy storage and transportation optimization, carbon sequestration process regulation and control and the like.

A method for performing core-to-log depth matching includes receiving input data. The input data includes core data and well log data. The method also includes performing an autonomous data preprocessing procedure to standardize the core data and the well log data to determine correlations between the core data and the well log data. The method also includes performing an autonomous outlier removal procedure to address differences in acquisition methods and measurement principles of the core data and the well log data. The method also includes automatically determining normalized cross-correlations between measurements derived from the core data and measurements derived from the well log data. The method also includes automatically shifting the measurements derived from the core data to a new depth position based upon a maximum of the normalized cross-correlations.

The invention relates to an electricity consumption information data cleaning and filtering method and system, and the method comprises the steps: dividing collected electricity consumption data into an end point data set, a middle data set and a tail end data set according to properties, and carrying out the basic abnormal value removal and missing data filling; converting the time domain data of each data set into a frequency domain pattern capable of representing the fluctuation mode of the time domain data through discrete Fourier transform; performing correlation analysis from top to bottom based on a graph, screening an intermediate data set by utilizing the graph of the endpoint data set, and classifying and screening an end data set by utilizing the screened intermediate data set graph; performing block processing on the screened end data set, extracting core features of the end data set by separating an input effect, a load effect and associated components, and dividing the core features into weak, medium and strong categories according to the fluctuation intensity of the features; carrying out inverse transformation on each processed graphic expression, and outputting cleaned and filtered high-quality time domain power utilization data; the accuracy of data cleaning can be improved, and the data value can be deeply mined.

This invention proposes an efficient system and method for processing objective data from helicopter flight simulators. The system includes a data preprocessing module for preprocessing acquired complete flight data, including data fusion and time-series correction; a data segmentation module for segmenting the raw simulator objective data based on helicopter configuration, control variable changes, and flight status to obtain segmented data; a batch processing module for providing a batch processing entry point for multiple sets of segmented data, simultaneously importing multiple sets of segmented data and multiple data parameters into one or more other processing modules for processing; an outlier removal module for identifying and removing outliers from the batch-processed segmented data; a curve fitting module for correcting the data after outlier removal; and a filtering module for recovering the true information as much as possible from the segmented data containing interference.

This invention discloses an industrial carbon emission detection and prediction method and system. It collects multi-source heterogeneous data from industrial production processes, and sequentially performs standardization, noise removal, outlier removal, and data fusion on the multi-source heterogeneous data to obtain initial carbon emission data. A hybrid detection and prediction model is constructed, employing an encoder to extract multi-factor correlation features from the multi-source data and a long short-term memory network to extract temporal fluctuation features of the carbon emission data. An attention mechanism is introduced to weight the correlation features and temporal fluctuation features. An improved IWOA whale optimization algorithm is used to optimize the hyperparameters of the hybrid detection and prediction model to obtain a target hybrid detection and prediction model. The initial carbon emission data is input into the target hybrid detection and prediction model, and the prediction results are output, including the current carbon emission detection value and the carbon emission prediction sequence. This improves the accuracy of carbon emission prediction results and the efficiency of carbon emission detection.

This invention discloses a method and system for detecting internal thread parameters based on laser point cloud analysis. Addressing the shortcomings of incomplete results and slow speed of traditional internal thread measurement methods, this device can quickly acquire point cloud information of the internal thread to be measured. Within the system, outlier removal, cylindrical fitting, and sawtooth wave fitting operations are performed on this point cloud information to determine the major diameter, minor diameter, pitch diameter, pitch, and thread angle parameters of the internal thread.

The invention provides a coal mine extremely shallow goaf geophysical prospecting data analysis method based on a transient electromagnetic method, which is characterized by comprising the following steps of: optimizing data acquisition parameters and correcting terrain; data preprocessing and noise suppression are carried out, multi-stage preprocessing is carried out on the original data, and preprocessing comprises outlier elimination, filtering noise reduction, null drift correction and background field deduction; time-depth conversion and resistivity inversion are carried out; quantitative analysis of goaf response characteristics; carrying out three-dimensional data fusion and visual modeling, integrating multi-measuring-line data and geological information, and constructing a goaf space distribution model; and the conclusion reliability is improved through field drilling verification and error source analysis. Aiming at the detection difficulty of an extremely shallow goaf, electromagnetic response details of a shallow medium are captured through small coil high-frequency excitation and microsecond-level signal acquisition. The terrain laser correction and the whole-period apparent resistivity inversion are combined, so that the tiny holes are accurately identified; and through drilling entity verification and error simulation, the misjudgment rate is remarkably reduced.

This invention relates to a method for generating visualized early warning information based on sensor data. The method includes: collecting multi-source sensor data from various UAVs, and performing outlier removal, missing value repair, and standardization to obtain preprocessed data; inputting the preprocessed data into a long short-term memory network to obtain risk values, predicted local state values, and early warning levels; generating sparse state features based on risk values, spatial proximity, and task area overlap, and uploading them to a collaborative decision-making center; updating the collaborative graph by the collaborative decision-making center to generate enhanced state representations and global states, further obtaining joint action values; predicting trajectories, calculating collision probabilities, generating visualized early warning information based on enhanced state representations and predicted local state values, and determining and issuing joint actions to relevant UAVs when action triggering conditions are met.

The present invention relates to a time series cross-calibration method and system for a synthetic aperture radar. The method comprises: screening a target area for time series cross-calibration; generating SAR image data pairs at each moment within a time range; extracting a median matrix of a data slice at a single moment using a window median method, obtaining a median matrix array, performing robust regression fitting with outlier removal, and generating a radiation calibration coefficient array; processing the radiation calibration coefficient array according to a final reference value of the radiation calibration coefficient, and obtaining a final radiation calibration coefficient. By performing robust regression fitting with outlier removal on the median matrix array, interference from outliers is avoided; calculating the radiation calibration coefficient corresponding to each single moment, using the correlation of the radiation calibration coefficients between different moments as a reference, combining the radiation calibration coefficient value measured at the moment to be calibrated and the radiation calibration coefficient value at the previous moment, and determining the final reference value of the radiation calibration coefficient, thereby improving calibration accuracy.

The application discloses a point cloud outlier removal method, a point cloud processing method, a device and related equipment. The method comprises the following steps: dividing a point cloud to be processed into multiple layers along a reference coordinate axis; determining whether to remove each point in each layer of the point cloud based on the bounding box of the point, and obtaining a second point cloud; obtaining the average nearest neighbor Euclidean distance of the second point cloud, that is, the nearest neighbor Euclidean distance of each point; and determining whether to remove each point in the second point cloud based on the difference between the nearest neighbor Euclidean distance of the point and the average nearest neighbor Euclidean distance, and the standard deviation of the nearest neighbor Euclidean distance of each point in the second point cloud and the average nearest neighbor Euclidean distance. Through the above steps, the application can remove the outliers in the point cloud more intuitively and conveniently while retaining the detailed features.

Systems and methods for iterative removal of outlier data from spectrometry data to determine one or more of a particle baseline and a detection threshold for nanoparticles are described. Ion signal intensity values that exceed an outlier threshold value associated with a sum of a first multiple of an average of the count distribution of ion signal intensity and a first multiple of a standard deviation of the count distribution of ion signal intensity are iteratively removed from the raw data set until no outliers remain, providing a background data set. A nanoparticle baseline intensity value is set as a sum of a second multiple of an average of the background data set and a second multiple of a standard deviation of the background data set to differentiate between signal intensity values that are associated with background interference and that are associated with the presence of nanoparticles in the sample.

The present invention discloses a method for removing outliers and generating reflectivity and normals based on multi-angle illumination images, which belongs to the fields of optics, digital image processing, self-supervised deep learning, etc. First, images under illumination from different directions are obtained; the images are read and normalized to obtain a multi-channel image array; a spatial coordinate system is established according to the position of the light source and the camera, and multiple light sources form a light source direction matrix; a neural network model is built, and a loss function is selected according to the characteristics of the outliers; each channel of the image array and the light source direction matrix are respectively input into the neural network model in pairs, and a back-propagation algorithm is used for iterative optimization to obtain a normal matrix with reflectivity scaling; the normal with reflectivity scaling is obtained by obtaining the vector modulus and normalizing it to obtain the reflectivity and normal of each channel, and the average value of the normal of each channel is the final normal. The present invention can be applied to solving the normal of an object, and can effectively reduce errors such as those caused by highlights, shadows, noise, etc.

This invention provides a zero-sample online fault diagnosis method based on a multi-resolution diffusion model. First, a source domain dataset U is constructed from data collected by a traction drive control simulation platform. S The target domain dataset U is constructed from data collected during actual train operation. T The fault category sets of the source and target domains do not overlap. A multi-resolution diffusion model is used to process the data, achieving outlier removal, missing data imputation, and standardization. A category-attribute description matrix A' is constructed, and fault features f from the source domain data are extracted using deep incremental supervised principal component analysis. i s Feature B s With attribute description a i The mapping relationship between them. Online acquisition of target domain data D. t Then, update the attribute descriptions of the fault categories and construct a new sparse matrix A. * The model parameters are then fine-tuned. Finally, Bayesian inference is used to perform real-time fault diagnosis on zero-sample data in the target domain, and the optimal fault category is inferred based on the maximum posterior probability, thus achieving accurate diagnosis and prediction of the target domain data.

The invention provides a data sampling and abnormal point removing method and system based on reinforcement learning. The method comprises the following steps: extracting multi-modal features, constructing corresponding relation features, extracting key points and descriptors thereof from two input images, generating an initial corresponding relation set, and performing coordinate normalization processing; constructing a reinforcement learning agent, and training the agent to enable the agent to learn an optimal strategy of data sampling and outlier removal; constructing a feature point relation graph, and dynamically segmenting normal matching points and outliers based on a clustering result represented by nodes and a confidence threshold; and carrying out multi-scale geometric verification on the screened high-confidence matching points, eliminating local inconsistent points, estimating an affine transformation matrix, and introducing illumination invariance constraint to optimize the image alignment precision. According to the technical scheme, through interaction with the environment, the reinforcement learning model can learn that the most representative strategy is selected under specific conditions, and the utilization efficiency of data and the performance of image matching are improved.

The invention belongs to the technical field of multi-channel spliced spectrum background deduction methods, and particularly relates to a laser plasma multi-channel spliced spectrum background deduction method, which comprises the following steps of: acquiring multi-channel original spectrum data, and performing preprocessing operations of wavelength resampling, noise filtering and abnormal point elimination on the original spectrum data to obtain a background deduction result; obtaining preprocessed spectral data; calculating multi-domain statistical characteristics based on the preprocessed spectral data, and generating a multi-domain auto-covariance structural element length field; the multi-domain statistical characteristics comprise the first-order difference, the second-order difference, the first-order difference variance, the second-order difference variance and the first-order and second-order difference covariance of the spectrum and the neighborhood of the spectrum; and performing multi-scale morphological opening operation on the preprocessed spectral data to obtain multi-scale upper and lower envelopes, determining a fusion weight based on a local amplitude unbalance degree, and fusing the upper and lower envelopes to obtain an initial morphological background envelope.

This invention provides a dynamic optimization scheduling method and system for green energy storage systems. The method collects operational data from the energy storage system, performs standardization processing and outlier removal to generate a standardized dataset. Based on this dataset, it extracts time-varying characteristics of environmental data, constructs a wind power prediction model, and considers the nonlinear characteristics of the energy storage system's charging and discharging efficiency and dynamic capacity changes to determine a feasible operating range. Combining the prediction model and the feasible range, it establishes a collaborative optimization objective function based on maximizing grid power smoothness, maximizing wind power absorption rate, and optimizing energy storage capacity utilization. A multi-objective optimization algorithm is used to solve the objective function, constructing a constraint set and establishing a state observation equation. Based on real-time data feedback, an adaptive control method is used to dynamically update the control input to achieve optimal scheduling. This invention can significantly improve prediction accuracy and operational efficiency, achieving multi-objective collaborative optimization and possessing significant application value.

An example apparatus is to clip a value of an activation associated with a layer of a floating-point version of a machine learning model to determine a clipped value of the activation, the value of the activation based on calibration data applied to the floating-point version of the machine learning model. The example apparatus is also to determine, using the clipped value of the activation, a quantization factor to quantize activations associated with a corresponding layer of a fixed-point version of the machine learning model. The example apparatus is further to configure the fixed-point version of the machine learning model on a device using the quantization factor.

This invention provides a substation equipment installation guidance and control method and system based on semantic perception and EKF algorithm, belonging to the field of substation equipment installation guidance and control technology. To address the technical problems of current substation equipment installation guidance methods relying on manual experience, resulting in poor installation accuracy, dependence on stringent auxiliary markers, and inability to adapt to unstructured installation environments, this invention preprocesses the raw point cloud data of the acquired equipment, including voxel downsampling and statistical outlier removal, filtering out dust and noise from the construction site. The unordered point cloud is classified into base, equipment, ground, or background, and then subjected to perception processing and semantic segmentation. In the segmented point cloud data, the flange plane is extracted using a random sampling consistency algorithm, and the bolt hole center coordinates are identified using a clustering algorithm. Coarse and fine registration strategies are employed for virtual-real registration operations. This invention is applied to substation equipment installation guidance.

The application relates to a multi-modal environment perception and post-fusion method of a sweeping robot, and relates to mobile robot environment perception. Geometric processing is first performed on depth point cloud data: voxel down-sampling, ROI cropping and outlier removal are performed on the point cloud by a CUDA parallel preprocessing unit, a ground surface is segmented by using a RANSAC algorithm with strong constraint of a gravity direction normal vector, and obstacle clustering is completed in combination with an Euclidean clustering algorithm; in parallel, two-dimensional semantic features are extracted from RGB image data by using a deep learning target detection model; the geometric and semantic information is input into a multi-modal cascade fusion and classification module, cross-modal space mapping is established, visual semantics is injected into geometric point cloud, and garbage and obstacles are accurately distinguished by using a multi-constraint cascade reasoning strategy; AED time sequence tracking and output are executed, a correlation measurement based on an aggregated Euclidean distance is introduced, real-time perception output with low delay and high robustness is realized on an embedded platform, and the problems of low obstacle detection and garbage misjudgment are solved.

This invention provides a time series data cleaning method and apparatus. The method includes: acquiring initial sequence data of an initial dataset, the initial sequence data including trend sequence data, periodic sequence data, and residual sequence data; performing a periodic component rationality judgment based on the initial sequence data to obtain target residual sequence data; performing anomaly detection on the target residual sequence data to obtain an outlier index of the initial dataset and performing an outlier removal operation to obtain the target dataset; and performing missing value imputation on the target dataset to complete the initial dataset cleaning. This invention uses a periodic component rationality judgment to analyze whether the periodic components of the sequence data are accurate and optimizes the residual sequence data. This significantly reduces the complexity of time series data while improving the accuracy of outlier identification during data cleaning, thus improving the accuracy, efficiency, and reliability of data cleaning, resulting in cleaner, more complete, and usable data after cleaning.

The invention relates to outlier removal for transformer network quantization. An example device will clip (494) a value of an activation associated with a layer of a floating point version of a machine learning model (410) to determine a clipped value of the activation, the value of the activation based on calibration data (455) applied to the floating point version of the machine learning model (410). The example device will also determine (496) a quantization factor (460) using the clipped value of the activation to quantify an activation associated with a corresponding layer of a fixed point version of the machine learning model (415). The example apparatus will further configure (425) the fixed point version of the machine learning model (415) on a device (430) using the quantization factor (460).

The invention discloses a cascade hydropower station data cleaning method, system and device based on the Pauta criterion and a medium, and relates to the technical field of water resource management. The method comprises the following steps: acquiring historical data, and preliminarily selecting a data cleaning method; the preliminarily selected data cleaning method comprises a manual selection method, an abnormal number proportion selection method and a twice Pauta criterion method; processing the historical data by using each data cleaning method to obtain three operation results, screening according to each operation result, and determining a two-time Pauta criterion method as a target data cleaning method; and processing target data by using the target data cleaning method, determining a data abnormal value, and performing abnormal value elimination and interpolation on the data abnormal value to complete cascade hydropower station data cleaning. According to the method, the problems of consistency, invalid values and missing values of the existing data set can be solved.

This invention relates to the fields of precision instrument manufacturing and precision testing and measurement technology, specifically a microstructure parameter measurement method based on a UNet network. First, addressing the presence of numerous prominent textures and scratches in the background region of the image of the test part, a region segmentation method based on a UNet network is employed to improve segmentation accuracy and overcome the influence of noise. Then, a dataset is collected and image enhancement is performed to improve the model's generalization ability. Horizontal and vertical flipping enriches the pose of the microstructure, and random variations in the HSV color gamut enhance the model's robustness to different lighting environments. After segmenting the target region, edge points are extracted and then fitted. Due to the presence of processing defects, outliers need to be removed before fitting. For micropore structures, outlier removal involves connecting edge sequence points to remove protrusions, and finally selecting the region with the largest connected component as the repaired feature region. For microgroove structures, outlier removal involves using the RANSAC algorithm for multiple iterations to filter out all interior points satisfying the "correct solution," and then using the data that best fits as input for least squares fitting, which can better estimate model parameters. This invention can improve the visual measurement accuracy of micro-holes and micro-grooves at the micrometer to millimeter level in workpieces.

This invention relates to the field of power load forecasting technology, specifically to a machine learning-based accurate power load forecasting system, comprising: a data processing module, which receives power load-related data through a multi-source data interface and performs data normalization, outlier removal, and missing value completion operations. This invention combines outlier removal and missing value completion to improve the quality of basic data and provide reliable data support for subsequent forecasts. Through multi-dimensional feature extraction and scientific screening, it comprehensively captures the temporal characteristics, correlation impact characteristics, and abrupt change characteristics of the load, strengthening the correlation between features and load and improving the model's generalization ability. By constructing a basic model library covering the entire time scale, it adaptively selects model combinations and dynamically allocates weights based on load data characteristics, improving the model's adaptability to different load fluctuation scenarios and enhancing the predictive specificity.

The invention discloses a deep learning-based radar echo data feature extraction and time sequence modeling method, and relates to the technical field of radar data processing, and the method comprises the steps: data preprocessing, outlier removal through an adaptive threshold, EMD and wavelet denoising, normalization through an improved formula, DS-CNN for multi-scale feature extraction, convolution with holes, and a capsule network. The GRU and a Transform encoder are combined through time sequence modeling, finally, attention is used for self-adaptive fusion of features, and feature representation is obtained through a ResNet full connection layer. By means of multiple advanced technologies, radar echo multi-scale and multi-mode characteristics are fully extracted, the time sequence relation is accurately modeled, model training is optimized, generalization and efficiency are improved, characteristics are balanced, radar data processing quality is comprehensively improved, and the application prospect is wide.

The invention relates to the technical field of molten iron temperature prediction around a ladle, and discloses a ladle molten iron temperature prediction model combining a genetic algorithm and a BP neural network, and the model comprises the steps: S1, screening out original index features according to expert experience and a metallurgical flow process mechanism; s2, processing missing values by adopting a direct deletion strategy, removing abnormal values by adopting a box plot method, and normalizing data; s3, on the basis of a classifier taking a random forest as a basic classifier, in combination with a Boruta algorithm of two tree-shaped classifiers XGboost and GBDT, carrying out importance sorting on original index features, determining an optimal feature number through cross validation, and carrying out feature screening by mutual coupling of the two features; and S4, introducing the screened index feature data, and pre-training the BP neural network structure by using GA. Through comparative analysis with prediction results of XGBoost and SVM models, the result shows that the GA-BP neural network shows more excellent performance in three evaluation indexes of RMSE, MAE and HR (%).

The invention discloses a drill rod thread area defect identification method and system based on three-dimensional point cloud deep learning. The method comprises the following steps: acquiring three-dimensional point cloud data of a drill rod thread area; performing statistical outlier removal, voxel downsampling, random resampling to preset fixed points, zero centralization and unit scale normalization processing on the point clouds in sequence to obtain standard point clouds; and inputting the standard point cloud into a pre-trained point cloud deep learning model for forward reasoning, and outputting a defect category and confidence. The point cloud deep learning model adopts a Point Net architecture and comprises an input level T-Net, and the T-Net is used for learning a spatial rigid transformation matrix and aligning input point clouds to reduce the influence of attitude difference on an identification result. And model training can be carried out based on the marked point cloud data set, and the engineering availability is improved in combination with visual rechecking.