32 results about "Data scrubbing" patented technology

The embodiment of the invention provides a video memory control method, device, equipment and system and a computer storage medium. The method comprises the steps that in response to an obtained drive upgrading request of an image processing unit (GPU), a before-upgrading page table corresponding to the GPU is obtained, and the before-upgrading page table comprises a mapping relation between a physical address and a virtual address of a GPU video memory; determining a data cleaning queue corresponding to the GPU, wherein the data cleaning queue is used for performing global cleaning on data in a GPU video memory; under the condition that the data cleaning queue is locked, driving upgrading operation is carried out on the GPU based on the driving upgrading request, an upgraded page table corresponding to the GPU is obtained, and the upgraded page table comprises a mapping relation between a physical address and a virtual address of a GPU video memory; and regulating and controlling data in the GPU video memory based on the pre-upgrading page table and the post-upgrading page table. In the embodiment of the invention, the driver upgrading operation of the GPU can be realized without operations such as video memory backup, shutdown, restart and the like.

The application discloses a ship carbon emission feature prediction method based on a dynamic method and an attention mechanism, adopts an AIS data-based dynamic method and a TFT time sequence prediction model based on the attention mechanism, utilizes ship AIS data, ship Lloyd's file data, pollutant emission factor parameters and geographic information, establishes a high-resolution ship carbon emission list, and realizes multi-period pollutant emission feature prediction of the ship. Through data cleaning, AIS data trajectory integration, data fusion and data completion and other operations, the AIS data is preprocessed, and the accuracy of the AIS data is ensured. According to the ship type and the ship running state, the dynamic method is used to estimate and analyze the pollutant emission amount, and the dynamic ship behavior is considered in real time when analyzing the carbon emission law and predicting the carbon emission feature.

This invention discloses a YOLO-based method for detecting surface defects on printed circuit boards, belonging to the fields of computer vision and intelligent manufacturing technology. The method includes the following steps: deploying an industrial area scan camera at the inspection station of an automated production line; when a PCB enters the inspection station, the camera is triggered by a sensor to acquire and store images; the acquired image data is processed using data cleaning, data augmentation, and target annotation methods; the processed PCB defect dataset undergoes detailed statistical analysis to optimize the model training strategy; based on the data analysis results, optimization is performed using the YOLOv5 target detection framework to establish a surface defect detection model suitable for PCB production scenarios. This invention can effectively improve the automation level and detection accuracy of defect detection, is suitable for modern large-scale batch production of PCBs, and has good engineering application prospects and industrial promotion value.

The invention relates to a bridge technical condition rating prediction method and system fusing maintenance process information. The method comprises the following steps: constructing a bridge maintenance process information data set based on a contour coefficient method and a K-means algorithm, and constructing a bridge regular inspection information database; carrying out discretization processing on the numeric data after data cleaning, and converting the numeric data into classified data; calculating Cramer's V coefficients between all the classification data and the bridge technical condition rating information, based on the selected feature data, converting the data with classification data attributes into one-hot codes, and adopting a principal component analysis method to reduce dimensionality to obtain dimensionality-reduced classification data; performing normalization and equalization processing to obtain a data set for model training; training a bridge technical condition rating prediction model by using a PSO-LightGBM algorithm; and predicting a bridge technical state rating level by using the prediction model. According to the invention, the accuracy of bridge technical condition rating prediction is improved.

The invention discloses an inactive user data management method, device and equipment and a storage medium, and relates to the technical field of data management, and the inactive user data management method comprises the following steps: configuring an inactive user data cleaning strategy, the data cleaning strategy defines a judgment condition of a user active state and at least one data cleaning operation corresponding to the judgment condition; periodically monitoring the active state of each user according to a preset time interval through the background service, and recognizing a target user meeting a data cleaning condition according to a judgment condition; and executing a data cleaning operation corresponding to the judgment condition on the target user to release the storage space. The problem of storage space waste of an Android operating system in a vehicle-mounted information entertainment system in a multi-user scene can be solved.

Embodiments of the present application provide a kind of processing method, device, electronic equipment, storage medium and product of abnormal data.The method comprises: obtaining the two-dimensional grid map to be processed;At least one specular reflection area is contained in the two-dimensional grid map;For any specular reflection area, determine a plurality of first line segments located in the specular reflection area, and determine the target boundary line segment of the specular reflection area based on the plurality of first line segments;Determine the spatial range of the specular reflection area based on the target boundary line segment, and delete all pixel points in the spatial range of the specular reflection area, provide a kind of systematic way to identify and remove abnormal data, reduce the need for manual intervention, support the data cleaning process of automation, and the spatial range of specular reflection area can be accurately determined, improve the cleaning efficiency and accuracy of abnormal data.

Embodiments of the present application provide a display memory control method, device, equipment, system and computer storage medium. The method comprises: in response to an obtained driving upgrade request of a graphics processing unit (GPU), obtaining a pre-upgrade page table corresponding to the GPU, the pre-upgrade page table comprising a mapping relationship between a physical address and a virtual address of GPU display memory; determining a data cleaning queue corresponding to the GPU, the data cleaning queue being used for globally cleaning data in the GPU display memory; under the condition of locking the data cleaning queue, performing a driving upgrade operation on the GPU based on the driving upgrade request, obtaining a post-upgrade page table corresponding to the GPU, the post-upgrade page table comprising a mapping relationship between the physical address and the virtual address of the GPU display memory; and based on the pre-upgrade page table and the post-upgrade page table, regulating the data in the GPU display memory. In the embodiments, the driving upgrade operation of the GPU can be implemented without the need for operations such as display memory backup, shutdown and restart.

The invention provides a hierarchical storage management method based on land observation satellite data value, which belongs to the field of application in a land observation satellite data processing system, and comprises the following steps of: calculating an initial comprehensive weight score during data archiving by creating seven types of core data tables such as a satellite parameter table and a data value parameter table; determining the number of storage days of the online or near-line cluster in combination with a dynamic storage strategy table; obtaining user order update data return times in real time, and recalculating the weight score at regular time to dynamically adjust the storage strategy; and finally, realizing data cleaning in combination with the water and fire wire threshold of the storage device. The problems of low resource utilization efficiency and slow user data return caused by making a storage strategy only based on the number of storage days in the prior art are solved, and the storage resource utilization rate and the user data response efficiency are remarkably improved.

A data lifecycle management method, device, equipment, storage medium and program product are disclosed, access heat of all data of a first storage layer in a data warehouse is acquired; wherein the data warehouse comprises at least two storage layers, and there is an association relationship between the storage layers, in which upstream data supports downstream data; cold data in the first storage layer is determined based on the access heat; associated data related to the cold data in the remaining storage layers is determined based on the association relationship; and a data cleaning operation is performed on the cold data and the associated data. By adopting the embodiment of the present application, the problems of hot data misdeletion and cold data redundancy in the data warehouse can be avoided, fine and differentiated management of data is realized, and resource waste of the data warehouse is reduced.

The application discloses a Redis cache elimination middleware based on a cuckoo filter, which is arranged between Redis and an application layer and comprises the following modules: a hot degree tracking module, a cold data cleaning module, an accurate elimination module, an adaptive tuning module, a memory management module and a concurrent control module. The application can realize intelligent elimination and dynamic management of a Redis cache space, and through accurate hot degree tracking, distributed cleaning, accurate elimination, adaptive tuning, intelligent memory management and efficient concurrent control, can effectively solve problems of low cache space utilization, long-term occupation of space by cold data and low elimination accuracy caused by a traditional cache elimination strategy, and improve the overall efficiency and resource utilization of a cache system.

The present disclosure relates to a bridge technical condition rating prediction method and system fusing maintenance history information. The method comprises: constructing a bridge maintenance history information dataset based on the profile coefficient method and the K-means algorithm, and constructing a bridge inspection information database; discretizing the numerical data after data cleaning to convert it into classification data; calculating the Cramer's V coefficient between all classification data and bridge technical condition rating information, selecting feature data based on this, converting data with classification data attributes into one-hot encoding, and reducing the dimension using the principal component analysis method to obtain reduced classification data; normalizing and equalizing to obtain a dataset for model training; training a bridge technical condition rating prediction model using the PSO-LightGBM algorithm; and predicting the bridge technical state rating level using the prediction model. The present disclosure improves the accuracy of bridge technical condition rating prediction.

The application discloses a data cleaning method and device, equipment, medium and product, comprising: displaying a data cleaning configuration control and M data table names corresponding to a first user identifier in a first configuration interface; receiving a first selection operation of selecting a first data table name and a second selection operation of the data cleaning configuration control in the first configuration interface; in response to the first selection operation and the second selection operation, displaying N fields in the first data table, and receiving a third selection operation of selecting a first field; in response to the third selection operation, determining that the first field is a cleaning basis field, and dividing the first data table into each data partition based on the cleaning basis field; displaying data distribution of each data partition in a second configuration interface, and receiving a cleaning range configuration operation in the second configuration interface; determining a cleaning range based on the cleaning range configuration operation, determining data in the first data table located in the cleaning range as to-be-cleaned data, and cleaning the to-be-cleaned data.

The invention relates to the technical field of data preprocessing, and provides a double-door sentry type entrance dirty data cleaning and error classification method and system. The double-door sentry type entrance dirty data cleaning and error classification method comprises the following steps: calculating the root mean square of each window; calculating spectrum dispersion; if the minimum value of the root-mean-square is smaller than the idle calibration value of the production line or the spectrum dispersion is larger than a spectrum dispersion threshold value, the event window is discarded; calculating a vibration peak value or a root-mean-square of the offline industrial sensing signal data, and if the vibration peak value or the root-mean-square exceeds a first threshold value, determining that the data is signal distortion data; performing consistency judgment on the sampling rate of the off-line industrial sensing signal data, and screening sampling rate error data; and calculating a cross correlation coefficient and an energy difference between the current event window and the previous frame event window, and if the cross correlation coefficient is smaller than a second threshold value and the energy difference is more than K times, judging that the data is packet loss data. The closed loop of training data self-cleaning and error reutilization in the offline environment of the local area network is realized.

The embodiment of the application provides a sandbox-based data cleaning method and system, electronic equipment and storage medium, and relates to the technical field of computers. The method is applied to a set application program in a user terminal, the set application program is in communication connection with a target application program running in a sandbox mechanism, and the method comprises the following steps: establishing a communication connection with a server; receiving a data cleaning instruction sent by the server, wherein the data cleaning instruction comprises the identification of the target application program and a to-be-cleaned data storage path, wherein the server sends the data cleaning instruction based on the cache data amount under each data storage path in the sandbox corresponding to the target application program acquired in real time; and sending the data cleaning instruction to the target application program based on the identification of the target application program, so that the target application program cleans the cache data under the to-be-cleaned data storage path. The application does not need the user to manually initiate the data cleaning instruction, and the cleaning efficiency and accuracy are relatively high.

The application discloses a kind of non-active user data management method, device and equipment and storage medium, it is related to data management technical field, non-active user data management method includes: configuration non-active user data cleaning strategy, data cleaning strategy defines the determination condition of user active state and at least one data cleaning operation corresponding to determination condition;Periodically monitor the active state of each user according to preset time interval by background service, according to determination condition, identify target user that satisfies data cleaning condition;The data cleaning operation corresponding to determination condition is executed to target user, to release storage space.The application can solve the waste problem of storage space in the multi-user scene of Android operating system in vehicle information entertainment system.

A method to perform data scrub operations by operating a memory device, the memory device comprising a main memory, an internal Error Correction Code (ECC) engine and a scrub memory. The method comprising: receiving a read command; accessing, based on the receiving the read command, a location in the main memory to read data at the location; error correcting data read during the accessing; and storing at the scrub memory information of the location based at least in part on the correcting the data meeting or exceeding an ECC threshold.

The application discloses an on-vehicle data cleaning method and device, electronic equipment and a storage medium, and aims at solving the technical problem of poor quality of the existing on-vehicle data cleaning method. The application comprises the following steps: reading a vehicle state log file; generating a filter table by using the automatic driving state filter rule table and the vehicle state log file; obtaining a bag file; filtering the bag file by using the filter table to obtain a target bag file, and deleting the bag file.

The invention relates to the technical field of industrial automation and intelligent control, and discloses a self-adaptive intelligent cleaning control method and system for a train open wagon, and the method comprises the steps: synchronously obtaining a material apparent image and tool-material interaction data when a target point is cleaned; obtaining an effect image after cleaning, calculating a residual rate, and deciding whether to reclean according to the residual rate; extracting and fusing based on the image and mechanical data to generate a multi-modal dynamic state feature vector; taking the feature vector as an input, taking an actual control parameter and a cleaning effect evaluation parameter as an output, constructing and continuously updating a time-varying inverse model on line, and adopting a model prediction control algorithm to solve an optimal control parameter vector of a next target point to be cleaned in a rolling manner; finally, the cleaning device is driven to execute self-adaptive cleaning, and a new round of control circulation is started. The cleaning device can automatically adapt to spatial changes of material characteristics, and intelligent global optimization of the cleaning effect, efficiency and energy consumption is achieved.

The present application provides a batch processing method for fiber static tensile test data, comprising: data reading and data processing, stress-strain data extraction, Young's modulus related calculation fitting point screening, linear fitting, data point marking and point graph output, data cleaning and origin processing, breaking strength, elongation and breaking work calculation, unit calibration and single group data output, cyclic processing and multi-group data operation, single group data Excel output, single Excel statistics and total Excel output. The present application solves the problem of lack of batch and accurate automatic processing method for fiber tensile machine data, significantly improves the efficiency and accuracy of processing batch data, is simple to operate, can be applied to various fiber tensile stress-strain data processing, and through the output of tensile stress-strain point graph, the data accuracy can be observed intuitively, effectively helps people to reduce the time cost consumption in the process of processing data, and provides a more efficient, accurate and user-friendly solution for fiber tensile test data processing.

The invention discloses a severe environment-oriented island power distribution network multi-modal data fusion and early warning method, which comprises the steps of obtaining multi-modal data in an island power distribution network operation process and performing data preprocessing, including data cleaning, de-noising and normalization processing, so as to obtain multi-modal normalized data of the same dimension; extracting equipment image features and equipment electrical operation features from the multi-modal normalized data, performing weighted fusion processing to obtain a fusion feature matrix, and performing equipment state recognition on the fusion feature matrix through an intelligent detection model; and when a fault condition is met, identifying a fault type and positioning a fault position to obtain a fault detection result of the island power distribution network equipment, carrying out fault early warning strategy matching of a corresponding risk level on the island power distribution network equipment according to the fault detection result, and executing the matched fault early warning strategy to carry out fault early warning. The method has the advantages of improving the accuracy and response speed of fault detection and effectively guaranteeing safe and stable operation of the island power distribution network.

The application discloses an accident classification method based on an LSTM-TextCNN algorithm combined with word cloud display, a storage medium and an apparatus, and the method comprises the following steps: (1) data mining; (2) determination of a table structure; (3) data preprocessing, including data cleaning, sentence segmentation, stop word removal processing and data warehousing; (4) classification training on accident information data stored in a database; (5) word cloud diagram display of accident classification information keywords by using a word cloud diagram method; and (6) back-end design: obtaining all-around accident information by establishing various services; the application can generate a word cloud diagram of accident classification keywords by using a deep learning algorithm combined with a word cloud technology, can very intuitively observe text keywords of various accident classifications, can directly obtain factors influencing accident classification by observing the word cloud diagram, can have a general understanding of a certain type of accident, and realizes visualization of the accident classification technology to a certain extent.

The invention discloses an intelligent management platform for the building industry. The intelligent management platform for the building industry comprises a personnel data deployment module, a data sensing module, a dynamic recognition module, a data entry module, a hotspot entry module, a data cleaning module and an information reminding module. The platform constructs an accident thermodynamic diagram through a hot spot input module, and intelligent pre-judgment of potential safety hazards of a construction site is achieved in combination with historical accident data and high-risk area information. The data cleaning module dynamically adjusts the priority and analysis precision of video identification according to the risk level of the thermodynamic diagram, improves the monitoring frame rate and background processing resources in a high-risk area, optimizes the distribution of system computing resources, avoids the problems of full-amount monitoring and low-efficiency analysis, and improves the accuracy and real-time performance of risk identification.

This disclosure relates to a method and system for data scrubbing in very-large-databases (VLDB) within regulated industries. The method involves segmenting a copy of a production database into manageable chunks based on row identifiers, allowing for parallel processing without database contention. Each chunk is processed by concurrently executing instances of a data scrubbing component, which replaces sensitive data with anonymous data of the same type using specified scrubbing algorithms. The system ensures that scrubbed data maintains the same structure and statistical properties as the original data while preventing the restoration of sensitive information. The process is meticulously documented to meet regulatory standards and facilitate audits, making it a robust solution for data scrubbing in environments with stringent compliance requirements.

The application relates to the technical field of big data, and particularly discloses a data cleaning method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: obtaining a historical cleaning cutoff time for cleaned data from historical cleaning parameters; determining a current cleaning start time for to-be-cleaned data based on the historical cleaning cutoff time; determining current cleaning parameters based on the historical cleaning parameters and the current cleaning start time; the current cleaning parameters comprise a plurality of sub cleaning parameters; determining to-be-cleaned partitions corresponding to each sub cleaning parameter according to a partition identifier in each sub cleaning parameter; and cleaning to-be-cleaned data in the to-be-cleaned partitions according to the current cleaning start time and the current cleaning days in the sub cleaning parameters. The method can improve the data cleaning efficiency.

A transformer chromatogram baseline anomaly identification method based on similarity and Wasserstein distance. The method comprises: first, performing preprocessing steps such as denoising by means of Gaussian filtering, baseline correction, data cleaning, data integration, data conversion and data reduction, so as to ensure data quality; then, establishing an anomaly-free standard chromatogram library, and extracting a baseline template; using a Pearson correlation coefficient to calculate the overall similarities between chromatograms to be measured and a standard chromatogram, and setting a similarity threshold value to preliminarily screen abnormal chromatograms; in addition, calculating the Wasserstein distances between baselines of the chromatograms to be measured and the standard baseline template, and setting a distance threshold value to accurately identify tiny anomalies such as baseline drift; by comprehensively determining the results of the similarities and the Wasserstein distances, determining whether there is a baseline anomaly in the chromatograms; and finally, developing a graphical user interface, outputting an anomaly identification result comprising the similarities, the Wasserstein distance values and anomaly types, and providing an anomaly alarm function. The present invention improves the accuracy and reliability of anomaly detection, and is suitable for maintenance and fault diagnosis of transformers.

The application discloses a high-altitude fireworks detection method based on an improved YOLOv5, and relates to the technical field of video detection, and in particular relates to a high-altitude fireworks detection method based on an improved YOLOv5. In the case of a plurality of high-altitude camera video streams, a multi-thread queue algorithm is applied, a frame extraction and uploading thread is set for each video stream, and the frame-extracted pictures of all threads are transmitted into an infinitely long queue. A high-altitude angle fireworks dataset is established, and data cleaning and dataset labeling work are carried out. The backbone network in the YOLOv5 network is modified, and each standard convolution layer after a CSP structure is replaced by a hybrid attention module. Training hyperparameters are set, the improved YOLOv5 network in the third step is trained based on the fireworks dataset established in the second step, a fireworks detection model is obtained after training is completed, and the pictures obtained in the first step are input into the fireworks detection model for smoke and flame detection. The improved target recognition algorithm is used to extract a single frame of a video stream to complete detection of fire abnormal phenomena.

The present application relates to the technical field of situation analysis display, and more particularly to a method and system for situation data processing and analysis based on time slicing, comprising the following steps: obtaining situation data from multiple sources, uniformly processing time information of the situation data to form a time reference; determining a time slicing window based on the time reference; performing time alignment processing on the situation data according to the time window to form time slicing data; performing data cleaning on the time slicing data to obtain a slicing data set; performing at least one kind of situation analysis processing based on the slicing data set, and organizing and displaying the analysis results according to the time reference to generate situation data output based on time slicing. The present application can reduce analysis deviation caused by inconsistent time and space references by uniformly processing multiple-source situation data in time and space and performing analysis based on time slicing, and can provide structured data support for situation research and decision-making.

This disclosure relates to a method and system for data scrubbing in very-large-databases (VLDB) within regulated industries. The method involves segmenting a copy of a production database into manageable chunks based on row identifiers, allowing for parallel processing without database contention. Each chunk is processed by concurrently executing instances of a data scrubbing component, which replaces sensitive data with anonymous data of the same type using specified scrubbing algorithms. The system ensures that scrubbed data maintains the same structure and statistical properties as the original data while preventing the restoration of sensitive information. The process is meticulously documented to meet regulatory standards and facilitate audits, making it a robust solution for data scrubbing in environments with stringent compliance requirements.

A system and method of scrubbing sensitive data from records using patterns and large language models (LLM). The method includes receiving a request to process a record comprising data including sensitive data. The method includes identifying, based on one or more regex rules, a first set of scrubbing candidates associated with the record. The method includes identifying, by a processing device and based on a large language model (LLM), a second set of scrubbing candidates associated with the record. The method includes generating, based on the first set of scrubbing candidates and the second set of scrubbing candidates, a scrubbed record by scrubbing the record to remove the sensitive data.

The invention discloses a delay sensitive business service quality situation prediction method and device, electronic equipment, a readable storage medium and a computer program product, and belongs to the technical field of information. The method comprises the following steps: acquiring operation log data of a base station operation and maintenance center (OMC) at the current moment; performing data cleaning and digital coding on the base station OMC operation log to obtain an OMC log body word vector matrix; performing feature extraction on the OMC log body word vector matrix to obtain a local feature vector and a long-distance feature vector, and performing feature fusion on the local feature vector and the long-distance feature vector to serve as a current input log feature vector; and inputting the current input log feature vector into a pre-constructed situation prediction neural network model, and obtaining a prediction result output by the situation prediction neural network model, the prediction result comprising a base station time delay sensitive business service quality situation scene of a preset duration after the current moment corresponding to the current input log feature vector.