133 results about "Data scrubbing" patented technology

A system and method of scrubbing sensitive data from records using patterns and large language models (LLM). The method includes receiving a request to process a record comprising data including sensitive data. The method includes identifying, based on one or more regex rules, a first set of scrubbing candidates associated with the record. The method includes identifying, by a processing device and based on a large language model (LLM), a second set of scrubbing candidates associated with the record. The method includes generating, based on the first set of scrubbing candidates and the second set of scrubbing candidates, a scrubbed record by scrubbing the record to remove the sensitive data.

The invention provides a cache data cleaning method and device, electronic equipment and a storage medium, and relates to the technical field of computer data storage.The method comprises the steps that a transition controller is used for replacing a fault controller for mirror image recombination, and a temporary mirror image pair is formed; it can be ensured that the basic data redundancy and the read-write function of the storage system are not seriously influenced; when the fault controller is recovered, the complex fault condition can be sensed in real time; once the concurrent fault occurs, the data cleaning process is adjusted according to execution state information of the data cleaning process, team information of the temporary mirror image pair and concurrent fault information of a concurrent fault controller; the data of the transition controller can be effectively cleaned under the condition of concurrent faults, and the fault-tolerant capability of the storage system is improved; when the cache data of the transition controller is cleaned, the overall performance and efficiency of the storage system are improved through an effective fault processing mechanism and accurate data cleaning process adjustment.

The invention provides a three-dimensional visualization expression method for a multi-scale geological structure model, including: collecting geological exploration data which includes geophysical information, drilling information, outcrop survey information, seismic information and geological age data; performing interpolation simulation on the geological exploration data to obtain an exploration sample set; preprocessing the exploration sample set to obtain preprocessed data, the preprocessing process including data cleaning, format unification and noise removal; constructing a multi-scale geological structure model according to the preprocessed data based on a three-dimensional modeling technology; performing scale decomposition on the geological structure model using a multi-scale analysis method to form model representations at different scale levels; and converting the model representations into 3D visualization images using a 3D graphics processing technology. The invention can effectively improve the expression ability of geological model and user experience, and has important practical application value for geological exploration and resource assessment.

The invention discloses a multi-source data fusion and intelligent scheduling method under a virtual power plant platform, relates to the technical field of virtual power plant optimal scheduling, and solves the technical problem that accurate adjustment of energy output is difficult to realize due to lack of a scientific energy scheduling priority evaluation system. According to the method, comprehensive data cleaning and deep fusion are carried out on the multi-source data, the data quality is improved, the potential correlation between the data is mined, the LSTM model and the fusion feature data are combined to establish the short-term load prediction model, the load change trend can be better captured, and the load prediction accuracy is improved. An energy scheduling priority evaluation system is constructed from three dimensions of policy scoring, economical efficiency scoring and adjustment capability scoring, comprehensive scoring and sorting are calculated through weighting, utilization of high-priority energy is guaranteed preferentially, and a refined scheduling strategy is formulated based on priority sorting and a load difference value when a load is short of supply or is over supply.

The invention provides a power distribution network photovoltaic power generation power prediction method based on WOA-VMD and a double-current network, and relates to the technical field of photovoltaic power generation, and the method comprises the steps: obtaining historical photovoltaic power generation power data and meteorological factors; performing data cleaning on the historical photovoltaic power generation power data; optimizing the VMD by using the WOA to obtain the optimal number of solid-state mode function components and decomposition penalty factors, and decomposing the preprocessed power generation power data by using the VMD to obtain the corresponding number of solid-state mode function components according to the optimal number of the solid-state mode function components and the decomposition penalty factors; taking the solid-state mode function components and the meteorological factors as input, and utilizing a pre-constructed BiGRU model and a pre-constructed GCN model to carry out prediction to obtain predicted values corresponding to the solid-state mode function components; and superposing the predicted values of the solid-state mode function components to obtain a final photovoltaic power generation power predicted value. According to the scheme, the problem that the photovoltaic power generation power prediction accuracy is low due to the fact that the original line load power is unstable and data relevance exists can be solved.

The invention relates to a system and method for low-voltage judgment of a power failure area based on mobile optical modem equipment, and the method comprises the steps: collecting the static data and offline event data of each mobile optical modem in real time, and storing the data after the data cleaning operation; based on historical offline event data of each mobile optical modem, an initial weight is distributed for each optical modem offline event, and then the offline weight of each mobile optical modem is dynamically adjusted according to the spatial and temporal distribution characteristics of the real-time offline event data of each mobile optical modem; in combination with the offline weight of each mobile optical modem, determining the power failure judgment weight of the corresponding area through space-time scanning statistics; and comparing the power failure judgment weight of each area with a corresponding power failure threshold value to judge whether a power failure fault occurs in the area, and if the power failure fault occurs, outputting position information and power failure warning information corresponding to the area. Compared with the prior art, the method can timely and accurately identify the regional power failure event and determine the fault region.

Post-write data management operations, such as refresh read, data scrub, and data relocation, are typically performed after a certain period of time has elapsed. However, performing such operations based on probability of access can provide advantages. So, in one example, a post-write data management operation is performed more frequently on relatively-warmer data than on relatively-colder data.

The invention provides a vector data processing method and system, and relates to the technical field of data processing, and the method comprises the steps: obtaining vector data, and carrying out the preprocessing of the vector data, and the preprocessing at least comprises the steps of data cleaning, data conversion and data topology restoration; establishing a spatial index for the preprocessed vector data; performing data compression and data fragmentation on the preprocessed vector data to obtain a plurality of vector data fragments; according to the method, the multiple vector data fragments are processed on the basis of the spatial index and the MapReduce framework, the processing efficiency of the vector data is improved through the distributed processing technology of the spatial index and the MapReduce framework, and the technical problem that existing vector data processing efficiency is low is solved.

The invention relates to a method, device and equipment for estimating the position of an elevator, a storage medium and a program product. The method comprises the steps that electric signal collection is conducted on the elevator traction machine, and voltage and current in a target time period are obtained; respectively determining the voltage frequency and the current frequency of the elevator traction machine according to the voltage and the current; the angular velocity of the elevator traction machine is calculated according to the voltage frequency, the current frequency and the number of pole pairs of the motor, and the comprehensive angular velocity is determined based on the voltage effective value corresponding to the voltage and the angular velocity; the start-stop time point of the elevator traction machine is determined, data cleaning is conducted on the comprehensive angular speed according to the start-stop time point, and the target angular speed is obtained; the target angular speed is input into an elevator system mathematical model for displacement calculation, and displacement information of the elevator car is obtained; and the position information of the elevator car is determined based on the displacement information and accumulated displacement information of the elevator car in the ascending or descending stage. By adopting the method, the estimation accuracy of the elevator position can be effectively improved.

The present disclosure provides a method, apparatus and system for monitoring ultra-high frequency partial discharge of a hydro-generator, and belongs to the technical field of hydro-generator partial discharge monitoring. The method includes: cleaning a partial discharge pulse sequence using a cleaning threshold to obtain a valid pulse sequence; performing redundant data filtering on each data unit divided from the valid pulse sequence to obtain a first target pulse sequence for short-period partial discharge monitoring; determining sub-sequences that are partial discharge events from the valid pulse sequence, forming a second target pulse sequence after associating an amplitude statistical feature of the sub-sequences, and storing the second target pulse sequence. The aforementioned method combines data cleaning, redundant data filtering, and partial discharge event identification, which enhances the real-time performance of partial discharge monitoring and records the long-period partial discharge change trend. The corresponding system adopts a multi-buffer zone and multi-processor architecture, thus further improving the real-time performance of partial discharge monitoring.

The embodiment of the invention provides a video memory control method, device, equipment and system and a computer storage medium. The method comprises the steps that in response to an obtained drive upgrading request of an image processing unit (GPU), a before-upgrading page table corresponding to the GPU is obtained, and the before-upgrading page table comprises a mapping relation between a physical address and a virtual address of a GPU video memory; determining a data cleaning queue corresponding to the GPU, wherein the data cleaning queue is used for performing global cleaning on data in a GPU video memory; under the condition that the data cleaning queue is locked, driving upgrading operation is carried out on the GPU based on the driving upgrading request, an upgraded page table corresponding to the GPU is obtained, and the upgraded page table comprises a mapping relation between a physical address and a virtual address of a GPU video memory; and regulating and controlling data in the GPU video memory based on the pre-upgrading page table and the post-upgrading page table. In the embodiment of the invention, the driver upgrading operation of the GPU can be realized without operations such as video memory backup, shutdown, restart and the like.

According to one embodiment of the present invention, a computer-implemented method for dynamically altering a frequency at which data scrubbing is performed on a memory device is disclosed. The computer-implemented method includes monitoring at least one of a temperature and a magnetic field of the memory device. The computer-implemented method further includes, responsive to determining that at least one of the temperature and the magnetic field of the memory device reaches and / or exceeds a predetermined threshold, respectively, increasing the frequency at which data scrubbing is performed on the memory device.

Sanitizing data can be a cumbersome task, particularly when the volume of data is large, the content is sensitive, and / or the type of sanitation requires contextual determinations. Sanitizing large amounts of data is tedious and may often require highly trained personnel with clearances and / or other qualifications. In the systems and methods of the present disclosure, language models (LMs) are used to solve these and other technical issues with tools that may allow sanitizing data easily, with high versatility, context awareness, and / or low demand for computational resources. In particular, some of the disclosed systems and methods use a first language model and a second language model (being less resource-intensive than the first language model) to generate sanitized output data with improved efficiency and accuracy. This dual-model approach ensures that sensitive information is handled appropriately while optimizing computer resource usage.

A transformer spectrogram baseline anomaly identification method based on similarity and Wasserstein distance comprises the following steps: firstly, through the preprocessing steps of Gaussian filtering denoising, baseline correction, data cleaning, data integration, data transformation, data reduction and the like, the data quality is ensured; then, establishing a standard chromatogram library without abnormity, and extracting a baseline template; and calculating the overall similarity between the spectrogram to be detected and the standard spectrogram by using a Pearson correlation coefficient, and setting a similarity threshold to preliminarily screen an abnormal spectrogram. Meanwhile, the Wasserstein distance between the to-be-measured spectrogram base line and the standard base line template is calculated, and a distance threshold value is set so as to accurately identify small anomalies such as base line drift and the like. And determining whether the baseline anomaly exists in the spectrogram or not by comprehensively judging the results of the similarity and the Wasserstein distance. And finally, developing a graphical user interface, outputting an anomaly recognition result, including similarity, a Wasserstein distance value and an anomaly type, and providing an anomaly alarm function. The method improves the accuracy and reliability of anomaly detection, and is suitable for transformer maintenance and fault diagnosis.

The invention discloses a personal voice intelligent terminal for a fund transaction room. The personal voice intelligent terminal comprises a voice data acquisition and classification module, a data cleaning and structuring module, a task concurrency feature extraction module, a concurrency situation prediction module, a situation classification module, a normal situation processing module and a high concurrency situation processing module. And the voice data acquisition and classification module is used for acquiring voice data of traders in real time through terminal equipment in a financial transaction scene. According to the method, the voice data is collected and classified in real time, the feature engineering and the deep learning model are combined, the voice stream processing mode is dynamically adjusted, the high task concurrency situation is effectively dealt with, and misrecognition and resource conflicts are reduced. Voice streams are segmented and fragments are identified independently under the conditions of resource saving and high concurrency in a normal situation, so that major identification errors are prevented, fund loss and strategy failure risks are reduced, the market order stability is guaranteed, and the efficiency and security of a transaction room are improved.

The invention relates to a method for generating an animation graphic (470) for at least one object (10), in particular a technical object such as a component or part, especially of a vehicle, for a digital presentation of the object (10), comprising the following steps: - Providing a CAD model (270) of the object (10); - Transforming (S20) the CAD model (270) into a user-defined CAD structure (340) using data cleansing software (320); - Generating a renderable and animatable data set (370) by tessellating and optimizing the CAD structure (340) using tessellation and optimization software (350); - Transferring the data set (370) into an animation module (400) using animation software (420); - Generating an animation graphic (470) using the animation software (420), wherein the animation software (420) uses selected animation parameters (450) that define a specific animation task for the digital presentation of the object (10).

The application discloses a ship carbon emission feature prediction method based on a dynamic method and an attention mechanism, adopts an AIS data-based dynamic method and a TFT time sequence prediction model based on the attention mechanism, utilizes ship AIS data, ship Lloyd's file data, pollutant emission factor parameters and geographic information, establishes a high-resolution ship carbon emission list, and realizes multi-period pollutant emission feature prediction of the ship. Through data cleaning, AIS data trajectory integration, data fusion and data completion and other operations, the AIS data is preprocessed, and the accuracy of the AIS data is ensured. According to the ship type and the ship running state, the dynamic method is used to estimate and analyze the pollutant emission amount, and the dynamic ship behavior is considered in real time when analyzing the carbon emission law and predicting the carbon emission feature.

The invention discloses an overlay-based traceless data cleaning method, which comprises the following steps of: after a file system is loaded when a system is started, creating an overlay file system, and overlapping and mounting contents of a root directory into the overlay file system according to a data protection partition range set by a user; data generated in the process of using the operating system by the user is stored in the overlay file system; and when data cleaning is needed, calling shred to perform physical layer deep cleaning on all data in the overlay file system. According to the method, the disk space does not need to be wasted for backup in advance, the file is backed up only when the file is changed, the disk space consumption is small (only the changed file is backed up), and the implementation (cleaning and restoring) time is short.

The invention discloses an airport parking apron target detection method based on Faster-RCNN, and belongs to the field of computer vision, and the method comprises the following steps: carrying out the continuous shooting of a parking apron region through an airport monitoring camera, and collecting image data, including various targets of airplanes, workers and ground service vehicles; processing the image data through methods of data cleaning, data enhancement, target labeling and the like; carrying out detailed statistical analysis on the processed data set to optimize a model training strategy; and optimization is carried out based on a Faster-RCNN target detection framework, and an airport parking apron target detection model is established. The target detection performance under the parking apron scene is improved, and therefore effective support is provided for airport operation management and safety monitoring.

A memory device includes a memory block including a plurality of sub-blocks each including a plurality of memory cells and control circuitry configured to perform single-side erase operations on the memory block in a sub-block mode in which a selected sub-block of the plurality of sub-blocks in the memory block is erased while unselected sub-blocks of the plurality of sub-blocks in the memory block are not erased and selectively perform data scrubbing and relocation operations on the plurality of sub-blocks of the memory block. To perform a data scrubbing and relocation operation, the control circuitry is configured to determine whether to perform the data scrubbing and relocation operation on a first sub-block based on a position of the first sub-block relative to an erase side of the memory block and selectively perform the data scrubbing and relocation operation on the first sub-block in response to the determination.

The application relates to a method, device, equipment, storage medium and program product for estimating the position of an elevator. The method comprises: collecting an electrical signal of an elevator hoisting machine to obtain voltage and current in a target period; determining the voltage frequency and current frequency of the elevator hoisting machine according to the voltage and current respectively; calculating the angular velocity of the elevator hoisting machine according to the voltage frequency, current frequency and motor pole pair number, and determining the comprehensive angular velocity based on the voltage effective value corresponding to the voltage and the angular velocity; determining the start-stop time point of the elevator hoisting machine, and performing data cleaning on the comprehensive angular velocity according to the start-stop time point to obtain a target angular velocity; inputting the target angular velocity into an elevator system mathematical model to perform displacement calculation and obtain displacement information of an elevator car; and determining the position information of the elevator car based on the displacement information and the cumulative displacement information of the elevator car in the current uplink or downlink stage. The method can effectively improve the estimation accuracy of the position of the elevator.

An embodiment of the present application provides a container scheduling method, device, and storage medium. In an embodiment of the present application, during the container scheduling process, for the first container where the target cloud application that the first user requests to exit is located, a cleanup operation can be performed on the first container according to a preset cleanup instruction to clean up the container data generated by the first user using the first container, and the association between the first container and the first user can be released to obtain a restored first container. On this basis, the restored first container can be used for subsequent container scheduling. Accordingly, in an embodiment of the present application, during the container scheduling process, the container data left after the user uses the container is cleaned up and the association between the container and the user is released through a preset cleanup instruction, which can achieve pre-preparation of container resources and no restart after use, thereby effectively reducing resource overhead in the container scheduling process.

This invention discloses a YOLO-based method for detecting surface defects on printed circuit boards, belonging to the fields of computer vision and intelligent manufacturing technology. The method includes the following steps: deploying an industrial area scan camera at the inspection station of an automated production line; when a PCB enters the inspection station, the camera is triggered by a sensor to acquire and store images; the acquired image data is processed using data cleaning, data augmentation, and target annotation methods; the processed PCB defect dataset undergoes detailed statistical analysis to optimize the model training strategy; based on the data analysis results, optimization is performed using the YOLOv5 target detection framework to establish a surface defect detection model suitable for PCB production scenarios. This invention can effectively improve the automation level and detection accuracy of defect detection, is suitable for modern large-scale batch production of PCBs, and has good engineering application prospects and industrial promotion value.

The invention relates to a bridge technical condition rating prediction method and system fusing maintenance process information. The method comprises the following steps: constructing a bridge maintenance process information data set based on a contour coefficient method and a K-means algorithm, and constructing a bridge regular inspection information database; carrying out discretization processing on the numeric data after data cleaning, and converting the numeric data into classified data; calculating Cramer's V coefficients between all the classification data and the bridge technical condition rating information, based on the selected feature data, converting the data with classification data attributes into one-hot codes, and adopting a principal component analysis method to reduce dimensionality to obtain dimensionality-reduced classification data; performing normalization and equalization processing to obtain a data set for model training; training a bridge technical condition rating prediction model by using a PSO-LightGBM algorithm; and predicting a bridge technical state rating level by using the prediction model. According to the invention, the accuracy of bridge technical condition rating prediction is improved.

The invention provides an enterprise complex decision auxiliary method and system based on a multi-modal cognitive intelligent large model, and belongs to the technical field of artificial intelligence. The method comprises the steps of collecting multi-modal enterprise complex decision auxiliary data, and performing data cleaning and data labeling to obtain a training data set; constructing an enterprise complex decision-making auxiliary large model; inputting the training data set into the enterprise complex decision auxiliary large model, and optimizing large model training to obtain a trained enterprise complex decision auxiliary large model; and deploying the trained large model to a server, and generating decision suggestions in combination with an input data enhancement strategy. According to the method, through joint processing of the bimodal data, the reasonability and the intelligent degree of model decision can be remarkably improved, and the model prediction accuracy in different scenes is ensured.

The embodiment of the present application provides a data cleaning method, comprising: when data cleaning is needed for a target disk included in a target server, determining whether a target control card for controlling the disk is integrated in the target server; when the target control card is integrated in the target server and the target control card has a data cleaning function, performing a first data cleaning operation on the target disk by calling the data cleaning function of the target control card; when it is determined that the first data cleaning operation is completed, modifying a configuration mode of the target disk to a JBOD mode, restarting the target server and mounting a target boot image, wherein the target boot image includes a first tool supporting data cleaning of the target disk; determining a target tool for data cleaning of the target disk according to a kind of the target disk according to a predetermined rule, and performing a second data cleaning operation on the target disk by using the target tool, wherein the first tool includes the target tool.

The invention discloses an inactive user data management method, device and equipment and a storage medium, and relates to the technical field of data management, and the inactive user data management method comprises the following steps: configuring an inactive user data cleaning strategy, the data cleaning strategy defines a judgment condition of a user active state and at least one data cleaning operation corresponding to the judgment condition; periodically monitoring the active state of each user according to a preset time interval through the background service, and recognizing a target user meeting a data cleaning condition according to a judgment condition; and executing a data cleaning operation corresponding to the judgment condition on the target user to release the storage space. The problem of storage space waste of an Android operating system in a vehicle-mounted information entertainment system in a multi-user scene can be solved.

Embodiments of the present application provide a kind of processing method, device, electronic equipment, storage medium and product of abnormal data.The method comprises: obtaining the two-dimensional grid map to be processed;At least one specular reflection area is contained in the two-dimensional grid map;For any specular reflection area, determine a plurality of first line segments located in the specular reflection area, and determine the target boundary line segment of the specular reflection area based on the plurality of first line segments;Determine the spatial range of the specular reflection area based on the target boundary line segment, and delete all pixel points in the spatial range of the specular reflection area, provide a kind of systematic way to identify and remove abnormal data, reduce the need for manual intervention, support the data cleaning process of automation, and the spatial range of specular reflection area can be accurately determined, improve the cleaning efficiency and accuracy of abnormal data.

The invention discloses a transformer substation three-dimensional modeling method based on a self-calibration technology, and relates to the technical field of transformer substation three-dimensional modeling, and the transformer substation three-dimensional modeling method based on the self-calibration technology comprises the following steps: S1, determining the spatial position of point cloud data; s2, coarse registration: performing coarse registration on the point cloud of the power transformation equipment without the specific mark and the point cloud of the power transformation equipment with the specific mark according to the relative position relation of the equipment; s3, data cleaning: performing cleaning and denoising processing on the point cloud to ensure data quality; s4, precise registration: fitting point cloud line and surface information for the point cloud of the power transformation equipment without the specific mark, and carrying out precise registration; according to the transformer substation three-dimensional modeling method based on the self-calibration technology, the calculation complexity can be effectively managed, efficient descriptors are used in the coarse registration stage to reduce the data volume, and therefore the overall registration speed is increased, and robustness can be improved.

Embodiments of the present application provide a display memory control method, device, equipment, system and computer storage medium. The method comprises: in response to an obtained driving upgrade request of a graphics processing unit (GPU), obtaining a pre-upgrade page table corresponding to the GPU, the pre-upgrade page table comprising a mapping relationship between a physical address and a virtual address of GPU display memory; determining a data cleaning queue corresponding to the GPU, the data cleaning queue being used for globally cleaning data in the GPU display memory; under the condition of locking the data cleaning queue, performing a driving upgrade operation on the GPU based on the driving upgrade request, obtaining a post-upgrade page table corresponding to the GPU, the post-upgrade page table comprising a mapping relationship between the physical address and the virtual address of the GPU display memory; and based on the pre-upgrade page table and the post-upgrade page table, regulating the data in the GPU display memory. In the embodiments, the driving upgrade operation of the GPU can be implemented without the need for operations such as display memory backup, shutdown and restart.