66 results about "Data scrubbing" patented technology

The invention discloses a multi-source data fusion and intelligent scheduling method under a virtual power plant platform, relates to the technical field of virtual power plant optimal scheduling, and solves the technical problem that accurate adjustment of energy output is difficult to realize due to lack of a scientific energy scheduling priority evaluation system. According to the method, comprehensive data cleaning and deep fusion are carried out on the multi-source data, the data quality is improved, the potential correlation between the data is mined, the LSTM model and the fusion feature data are combined to establish the short-term load prediction model, the load change trend can be better captured, and the load prediction accuracy is improved. An energy scheduling priority evaluation system is constructed from three dimensions of policy scoring, economical efficiency scoring and adjustment capability scoring, comprehensive scoring and sorting are calculated through weighting, utilization of high-priority energy is guaranteed preferentially, and a refined scheduling strategy is formulated based on priority sorting and a load difference value when a load is short of supply or is over supply.

The invention relates to a system and method for low-voltage judgment of a power failure area based on mobile optical modem equipment, and the method comprises the steps: collecting the static data and offline event data of each mobile optical modem in real time, and storing the data after the data cleaning operation; based on historical offline event data of each mobile optical modem, an initial weight is distributed for each optical modem offline event, and then the offline weight of each mobile optical modem is dynamically adjusted according to the spatial and temporal distribution characteristics of the real-time offline event data of each mobile optical modem; in combination with the offline weight of each mobile optical modem, determining the power failure judgment weight of the corresponding area through space-time scanning statistics; and comparing the power failure judgment weight of each area with a corresponding power failure threshold value to judge whether a power failure fault occurs in the area, and if the power failure fault occurs, outputting position information and power failure warning information corresponding to the area. Compared with the prior art, the method can timely and accurately identify the regional power failure event and determine the fault region.

The present disclosure provides a method, apparatus and system for monitoring ultra-high frequency partial discharge of a hydro-generator, and belongs to the technical field of hydro-generator partial discharge monitoring. The method includes: cleaning a partial discharge pulse sequence using a cleaning threshold to obtain a valid pulse sequence; performing redundant data filtering on each data unit divided from the valid pulse sequence to obtain a first target pulse sequence for short-period partial discharge monitoring; determining sub-sequences that are partial discharge events from the valid pulse sequence, forming a second target pulse sequence after associating an amplitude statistical feature of the sub-sequences, and storing the second target pulse sequence. The aforementioned method combines data cleaning, redundant data filtering, and partial discharge event identification, which enhances the real-time performance of partial discharge monitoring and records the long-period partial discharge change trend. The corresponding system adopts a multi-buffer zone and multi-processor architecture, thus further improving the real-time performance of partial discharge monitoring.

The embodiment of the invention provides a video memory control method, device, equipment and system and a computer storage medium. The method comprises the steps that in response to an obtained drive upgrading request of an image processing unit (GPU), a before-upgrading page table corresponding to the GPU is obtained, and the before-upgrading page table comprises a mapping relation between a physical address and a virtual address of a GPU video memory; determining a data cleaning queue corresponding to the GPU, wherein the data cleaning queue is used for performing global cleaning on data in a GPU video memory; under the condition that the data cleaning queue is locked, driving upgrading operation is carried out on the GPU based on the driving upgrading request, an upgraded page table corresponding to the GPU is obtained, and the upgraded page table comprises a mapping relation between a physical address and a virtual address of a GPU video memory; and regulating and controlling data in the GPU video memory based on the pre-upgrading page table and the post-upgrading page table. In the embodiment of the invention, the driver upgrading operation of the GPU can be realized without operations such as video memory backup, shutdown, restart and the like.

Sanitizing data can be a cumbersome task, particularly when the volume of data is large, the content is sensitive, and / or the type of sanitation requires contextual determinations. Sanitizing large amounts of data is tedious and may often require highly trained personnel with clearances and / or other qualifications. In the systems and methods of the present disclosure, language models (LMs) are used to solve these and other technical issues with tools that may allow sanitizing data easily, with high versatility, context awareness, and / or low demand for computational resources. In particular, some of the disclosed systems and methods use a first language model and a second language model (being less resource-intensive than the first language model) to generate sanitized output data with improved efficiency and accuracy. This dual-model approach ensures that sensitive information is handled appropriately while optimizing computer resource usage.

The application discloses a ship carbon emission feature prediction method based on a dynamic method and an attention mechanism, adopts an AIS data-based dynamic method and a TFT time sequence prediction model based on the attention mechanism, utilizes ship AIS data, ship Lloyd's file data, pollutant emission factor parameters and geographic information, establishes a high-resolution ship carbon emission list, and realizes multi-period pollutant emission feature prediction of the ship. Through data cleaning, AIS data trajectory integration, data fusion and data completion and other operations, the AIS data is preprocessed, and the accuracy of the AIS data is ensured. According to the ship type and the ship running state, the dynamic method is used to estimate and analyze the pollutant emission amount, and the dynamic ship behavior is considered in real time when analyzing the carbon emission law and predicting the carbon emission feature.

The invention discloses an airport parking apron target detection method based on Faster-RCNN, and belongs to the field of computer vision, and the method comprises the following steps: carrying out the continuous shooting of a parking apron region through an airport monitoring camera, and collecting image data, including various targets of airplanes, workers and ground service vehicles; processing the image data through methods of data cleaning, data enhancement, target labeling and the like; carrying out detailed statistical analysis on the processed data set to optimize a model training strategy; and optimization is carried out based on a Faster-RCNN target detection framework, and an airport parking apron target detection model is established. The target detection performance under the parking apron scene is improved, and therefore effective support is provided for airport operation management and safety monitoring.

The application relates to a method, device, equipment, storage medium and program product for estimating the position of an elevator. The method comprises: collecting an electrical signal of an elevator hoisting machine to obtain voltage and current in a target period; determining the voltage frequency and current frequency of the elevator hoisting machine according to the voltage and current respectively; calculating the angular velocity of the elevator hoisting machine according to the voltage frequency, current frequency and motor pole pair number, and determining the comprehensive angular velocity based on the voltage effective value corresponding to the voltage and the angular velocity; determining the start-stop time point of the elevator hoisting machine, and performing data cleaning on the comprehensive angular velocity according to the start-stop time point to obtain a target angular velocity; inputting the target angular velocity into an elevator system mathematical model to perform displacement calculation and obtain displacement information of an elevator car; and determining the position information of the elevator car based on the displacement information and the cumulative displacement information of the elevator car in the current uplink or downlink stage. The method can effectively improve the estimation accuracy of the position of the elevator.

An embodiment of the present application provides a container scheduling method, device, and storage medium. In an embodiment of the present application, during the container scheduling process, for the first container where the target cloud application that the first user requests to exit is located, a cleanup operation can be performed on the first container according to a preset cleanup instruction to clean up the container data generated by the first user using the first container, and the association between the first container and the first user can be released to obtain a restored first container. On this basis, the restored first container can be used for subsequent container scheduling. Accordingly, in an embodiment of the present application, during the container scheduling process, the container data left after the user uses the container is cleaned up and the association between the container and the user is released through a preset cleanup instruction, which can achieve pre-preparation of container resources and no restart after use, thereby effectively reducing resource overhead in the container scheduling process.

This invention discloses a YOLO-based method for detecting surface defects on printed circuit boards, belonging to the fields of computer vision and intelligent manufacturing technology. The method includes the following steps: deploying an industrial area scan camera at the inspection station of an automated production line; when a PCB enters the inspection station, the camera is triggered by a sensor to acquire and store images; the acquired image data is processed using data cleaning, data augmentation, and target annotation methods; the processed PCB defect dataset undergoes detailed statistical analysis to optimize the model training strategy; based on the data analysis results, optimization is performed using the YOLOv5 target detection framework to establish a surface defect detection model suitable for PCB production scenarios. This invention can effectively improve the automation level and detection accuracy of defect detection, is suitable for modern large-scale batch production of PCBs, and has good engineering application prospects and industrial promotion value.

The invention relates to a bridge technical condition rating prediction method and system fusing maintenance process information. The method comprises the following steps: constructing a bridge maintenance process information data set based on a contour coefficient method and a K-means algorithm, and constructing a bridge regular inspection information database; carrying out discretization processing on the numeric data after data cleaning, and converting the numeric data into classified data; calculating Cramer's V coefficients between all the classification data and the bridge technical condition rating information, based on the selected feature data, converting the data with classification data attributes into one-hot codes, and adopting a principal component analysis method to reduce dimensionality to obtain dimensionality-reduced classification data; performing normalization and equalization processing to obtain a data set for model training; training a bridge technical condition rating prediction model by using a PSO-LightGBM algorithm; and predicting a bridge technical state rating level by using the prediction model. According to the invention, the accuracy of bridge technical condition rating prediction is improved.

The embodiment of the present application provides a data cleaning method, comprising: when data cleaning is needed for a target disk included in a target server, determining whether a target control card for controlling the disk is integrated in the target server; when the target control card is integrated in the target server and the target control card has a data cleaning function, performing a first data cleaning operation on the target disk by calling the data cleaning function of the target control card; when it is determined that the first data cleaning operation is completed, modifying a configuration mode of the target disk to a JBOD mode, restarting the target server and mounting a target boot image, wherein the target boot image includes a first tool supporting data cleaning of the target disk; determining a target tool for data cleaning of the target disk according to a kind of the target disk according to a predetermined rule, and performing a second data cleaning operation on the target disk by using the target tool, wherein the first tool includes the target tool.

The invention discloses an inactive user data management method, device and equipment and a storage medium, and relates to the technical field of data management, and the inactive user data management method comprises the following steps: configuring an inactive user data cleaning strategy, the data cleaning strategy defines a judgment condition of a user active state and at least one data cleaning operation corresponding to the judgment condition; periodically monitoring the active state of each user according to a preset time interval through the background service, and recognizing a target user meeting a data cleaning condition according to a judgment condition; and executing a data cleaning operation corresponding to the judgment condition on the target user to release the storage space. The problem of storage space waste of an Android operating system in a vehicle-mounted information entertainment system in a multi-user scene can be solved.

Embodiments of the present application provide a kind of processing method, device, electronic equipment, storage medium and product of abnormal data.The method comprises: obtaining the two-dimensional grid map to be processed;At least one specular reflection area is contained in the two-dimensional grid map;For any specular reflection area, determine a plurality of first line segments located in the specular reflection area, and determine the target boundary line segment of the specular reflection area based on the plurality of first line segments;Determine the spatial range of the specular reflection area based on the target boundary line segment, and delete all pixel points in the spatial range of the specular reflection area, provide a kind of systematic way to identify and remove abnormal data, reduce the need for manual intervention, support the data cleaning process of automation, and the spatial range of specular reflection area can be accurately determined, improve the cleaning efficiency and accuracy of abnormal data.

Embodiments of the present application provide a display memory control method, device, equipment, system and computer storage medium. The method comprises: in response to an obtained driving upgrade request of a graphics processing unit (GPU), obtaining a pre-upgrade page table corresponding to the GPU, the pre-upgrade page table comprising a mapping relationship between a physical address and a virtual address of GPU display memory; determining a data cleaning queue corresponding to the GPU, the data cleaning queue being used for globally cleaning data in the GPU display memory; under the condition of locking the data cleaning queue, performing a driving upgrade operation on the GPU based on the driving upgrade request, obtaining a post-upgrade page table corresponding to the GPU, the post-upgrade page table comprising a mapping relationship between the physical address and the virtual address of the GPU display memory; and based on the pre-upgrade page table and the post-upgrade page table, regulating the data in the GPU display memory. In the embodiments, the driving upgrade operation of the GPU can be implemented without the need for operations such as display memory backup, shutdown and restart.

The present disclosure describes methods, system, and devices for data processing in a mobile communication system. The method includes receiving, by a base station, data cleaning requirement for preprocessing measurement data from a first node providing control functions; receiving, by the base station, reference signal from a user equipment (UE); obtaining, by the base station, measurement data based on the reference signal; preprocessing, by the base station, the measurement data to obtain the preprocessed measurement data according to the data cleaning requirement; and sending, by the base station, the preprocessed measurement data to a second node for next step processing.

The invention provides an LDL-C concentration estimation method based on an interpretability enhancement machine, and belongs to the technical field of medical detection and artificial intelligence. The LDL-C concentration estimation method comprises the following steps: selecting a subject with a complete blood fat detection record from a preset database to obtain initial data; performing data cleaning on the initial data, and longitudinally merging the blood fat measurement data under the two visits to obtain a data set; performing regression modeling on the EBM based on the training set and the test set to obtain an EBM model; calculating a contribution value of each input ratio variable by constructing a square view of a lookup table based on an EBM model, adding all the contribution values, and calculating a final LDL-C predicted value through a connection function g; and performing comprehensive evaluation and optimization on the EBM model by adopting a plurality of statistical indexes, and integrating the EBM model into various medical systems. Through high-quality data, an optimized data processing strategy, an accurate EBM prediction model, an interpretable prediction result and efficient system integration, accurate estimation and wide application of LDL-C are realized, and the method is superior to a traditional formula calculation method.

The invention provides a hierarchical storage management method based on land observation satellite data value, which belongs to the field of application in a land observation satellite data processing system, and comprises the following steps of: calculating an initial comprehensive weight score during data archiving by creating seven types of core data tables such as a satellite parameter table and a data value parameter table; determining the number of storage days of the online or near-line cluster in combination with a dynamic storage strategy table; obtaining user order update data return times in real time, and recalculating the weight score at regular time to dynamically adjust the storage strategy; and finally, realizing data cleaning in combination with the water and fire wire threshold of the storage device. The problems of low resource utilization efficiency and slow user data return caused by making a storage strategy only based on the number of storage days in the prior art are solved, and the storage resource utilization rate and the user data response efficiency are remarkably improved.

The invention relates to the technical field of data cleaning, in particular to a data cleaning structure for CMOS data cleaning and a mainboard. The data cleaning structure comprises a body, an elastic piece and a trigger key; the body comprises fulcrums; the elastic piece has a set length, a first contact pin, a second contact pin and a third contact pin are arranged along the length direction of the elastic piece, a support position is arranged on the elastic piece between the second contact pin and the third contact pin, and the elastic piece is erected on the fulcrum through the support position; the trigger key comprises an abutting end, and the abutting end faces the third contact pin and is used for triggering or releasing triggering of the third contact pin; when the third contact pin is released from triggering, the first contact pin and the second contact pin are used for being in contact with a first contact piece and a second contact piece on the mainboard respectively; when the trigger key is triggered, the second contact pin and the third contact pin are used for being in contact with the second contact piece and the third contact piece on the mainboard respectively. The operation is simple and quick, the loss of parts can be avoided, and the situation of misoperation is reduced.

The invention relates to a line loss electric quantity analysis-based abnormal metering electric energy meter searching method, which comprises the following steps of: firstly, selecting a line of which the line loss rate exceeds a normal range, after confirming information of all node electric energy meters of the line, extracting daily frozen electric quantity readings and 15min frozen current data of all the electric energy meters on the line within a period of time, and performing data cleaning; calculating daily line loss electric quantity of the line and a daily current summation value of each node electric energy meter on the line by using the cleaned data, identifying and deleting an abnormal value according to a Pearson criterion, and then calculating a pearson correlation coefficient between the line loss electric quantity of the line and the summation current of each node electric energy meter; when the correlation coefficient between the current summation value of the node n electric energy meter and the line loss electric quantity is larger than 0.7, it can be judged that strong correlation exists between the current summation value of the node n electric energy meter and the overproof line loss electric quantity, and it is judged that metering abnormity exists in the node n electric energy meter according to the strong correlation. According to the invention, the capability and the level of metering abnormity online detection are effectively improved.

A data lifecycle management method, device, equipment, storage medium and program product are disclosed, access heat of all data of a first storage layer in a data warehouse is acquired; wherein the data warehouse comprises at least two storage layers, and there is an association relationship between the storage layers, in which upstream data supports downstream data; cold data in the first storage layer is determined based on the access heat; associated data related to the cold data in the remaining storage layers is determined based on the association relationship; and a data cleaning operation is performed on the cold data and the associated data. By adopting the embodiment of the present application, the problems of hot data misdeletion and cold data redundancy in the data warehouse can be avoided, fine and differentiated management of data is realized, and resource waste of the data warehouse is reduced.

The application discloses a Redis cache elimination middleware based on a cuckoo filter, which is arranged between Redis and an application layer and comprises the following modules: a hot degree tracking module, a cold data cleaning module, an accurate elimination module, an adaptive tuning module, a memory management module and a concurrent control module. The application can realize intelligent elimination and dynamic management of a Redis cache space, and through accurate hot degree tracking, distributed cleaning, accurate elimination, adaptive tuning, intelligent memory management and efficient concurrent control, can effectively solve problems of low cache space utilization, long-term occupation of space by cold data and low elimination accuracy caused by a traditional cache elimination strategy, and improve the overall efficiency and resource utilization of a cache system.

There is provided a technique that contributes to the optimization of data cleaning prior to data analysis. A method for optimizing data processing of an input table includes performing a limitation process in which an extraction table is generated from a first target table using bits of a null bitmap by extracting a column having one or more missing values, the null bitmap being a metadata of the first target table; and performing a count process in the extraction table, in the count process the number of missing values per row of the extraction table being obtained by performing a calculation with the bits of the null bitmap, wherein, the limitation process is performed with the input table as the first target table.

The invention discloses a curve fitting-based yin and yang circle segmentation method, which belongs to the field of data processing, solves the problem that the traditional data processing method often cannot accurately distinguish the internal diameter data of a yin line and a yang line, and comprises the following steps of: data acquisition: performing high-frequency scanning on the surface of an inner bore of a gun / gun barrel by adopting an optical sensor, obtaining inner bore original data of the yin and yang lines; data preprocessing: performing data cleaning and smoothing processing on the original data to remove abnormal values and noise; curve fitting: performing polynomial fitting and interpolation on the processed data to obtain a segmentation threshold curve; curve segmentation: utilizing a segmentation threshold curve to segment internal diameter data of a negative line and a positive line; and data output: outputting the inner diameter data of the divided yin and yang lines for further analysis and use. According to the method, the inner diameter data of the negative line and the positive line can be accurately distinguished, and necessary invalid data removal and noise reduction requirements are provided.

The present disclosure relates to a bridge technical condition rating prediction method and system fusing maintenance history information. The method comprises: constructing a bridge maintenance history information dataset based on the profile coefficient method and the K-means algorithm, and constructing a bridge inspection information database; discretizing the numerical data after data cleaning to convert it into classification data; calculating the Cramer's V coefficient between all classification data and bridge technical condition rating information, selecting feature data based on this, converting data with classification data attributes into one-hot encoding, and reducing the dimension using the principal component analysis method to obtain reduced classification data; normalizing and equalizing to obtain a dataset for model training; training a bridge technical condition rating prediction model using the PSO-LightGBM algorithm; and predicting the bridge technical state rating level using the prediction model. The present disclosure improves the accuracy of bridge technical condition rating prediction.

The application discloses a high-dimensional soil data visualization method and system based on density peak clustering, and the visualization method comprises the following steps: S1, a data set comprising a plurality of samples is acquired, and data cleaning and standardization processing are performed on the samples in the data set, and each sample comprises various types of rock-soil data; S2, the data set after the standardization processing is labeled according to the geographical positions of stations, and then dimension reduction processing is performed on the data set after the label labeling, so as to obtain 3-dimensional reduced data; S3, a density peak clustering algorithm is used to cluster the 3-dimensional reduced data, so as to obtain a clustering result; and S4, the visualization method is used to convert the clustering result into a graphic display.

The present application belongs to the technical field of data processing, and specifically relates to a multi-source data analysis and evaluation method for constructing a jogging index, and specifically comprises the following steps: obtaining complete road network information and performing data cleaning and optimization on the same; superimposing movement trajectory data on the cleaned road network data, capturing the distribution of the trajectory data on each road segment, and calculating the jogging frequency; extracting the subjective and objective environmental related features respectively; establishing a random forest regression model, optimizing the key parameters of the random forest regression model, exploring the nonlinear relationship between the features and the jogging frequency, and obtaining the threshold value of each feature; and constructing a jogging index for evaluating the jogging capacity of urban streets. The present application overcomes the shortcomings of the prior art, integrates multi-source data, uses a random forest model to deeply explore the key influencing factors and the internal mechanism of the jogging capacity of urban streets, and provides technical support for scientifically evaluating the jogging suitability of streets.

The application discloses a data cleaning method and device, equipment, medium and product, comprising: displaying a data cleaning configuration control and M data table names corresponding to a first user identifier in a first configuration interface; receiving a first selection operation of selecting a first data table name and a second selection operation of the data cleaning configuration control in the first configuration interface; in response to the first selection operation and the second selection operation, displaying N fields in the first data table, and receiving a third selection operation of selecting a first field; in response to the third selection operation, determining that the first field is a cleaning basis field, and dividing the first data table into each data partition based on the cleaning basis field; displaying data distribution of each data partition in a second configuration interface, and receiving a cleaning range configuration operation in the second configuration interface; determining a cleaning range based on the cleaning range configuration operation, determining data in the first data table located in the cleaning range as to-be-cleaned data, and cleaning the to-be-cleaned data.

The invention relates to the technical field of traffic management, in particular to a method for etching a traffic management level based on alarm data of alarm conditions, police forces and road abnormal events, which comprises the following steps of: acquiring alarm condition data, police force data, traffic abnormal event data and the total number of motor vehicles from corresponding data sources through a supercomputing center; after data cleaning, data comparison, data conversion and data analysis, a traffic abnormal event factor index, an alarm condition factor index, a police force factor index, a traffic abnormal event and police force matching degree index, an alarm condition and police force matching degree index and a police force execution efficiency index are calculated according to the traffic management level description model; calculating a traffic management comprehensive index according to each index and the corresponding weight; and according to the score range of the traffic management comprehensive index, performing grade evaluation on the traffic management level and generating a corresponding police force allocation scheme. The system can accurately evaluate the traffic management level and generate a police force allocation scheme, and efficiently optimizes the traffic management.