217 results about "Dirty data" patented technology

Certain example embodiments relate to dashboards that help streamline and automate data quality management processes used with machine learning (ML) models and ML-enabled technology. A clean dataset is initialized from a dirty dataset. A search space is the set of all possible combinations of available error detection algorithms and data repair algorithms. A scoring function measures performance of a given error detection algorithm and data repair algorithm combination on the clean dataset. An ML model is trained using the clean dataset. Best error detection and data repair algorithms are selected, based on an optimization on the set of all possible combinations, and the defined scoring function. The selected best error detection algorithm is applied to the clean dataset, and a repaired dataset is generated using the selected best repair algorithm. The clean dataset is set to the repaired dataset. This procedure is repeated until a condition is met.

The invention discloses a cache space control method of a storage system, electronic equipment and a storage medium, and relates to the technical field of data caching, and the method comprises the steps of identifying dirty data, comparing the size relation between the hit rate of cached data in a cache space and a first hit rate threshold value and a second hit rate threshold value, and when the hit rate is greater than the first hit rate threshold value, controlling the cache space of the storage system. When the hit rate is smaller than a first hit rate threshold value, executing an expansion instruction to control the cache module to expand a cache space, and when the hit rate is smaller than a second hit rate threshold value, controlling the cache module to brush dirty data down to a rear-end memory, executing a reduction instruction to control the cache module to reduce the cache space, and performing a cache hit rate judgment mechanism and a linkage telescopic cache mechanism in a full random access scene. The problems of performance reduction, high delay of data access, high power consumption and the like of the cache system due to the fact that the cache space is fixed and cannot adapt to different application scenes and changes of data loads in the prior art are solved, performance self-adaption of the IO model is achieved, the response speed is increased, and the performance of the cache system in multiple storage scenes is improved.

The invention relates to the technical field of artificial intelligence, in particular to an electrical connection safety management and control method based on intelligent monitoring, which comprises the steps of data acquisition, time synchronization, quality evaluation, feature engineering and the like. According to the method, electric, thermal, mechanical, acoustic and other channels are placed on the same time axis through unified alignment and quality marking of multi-modal data and two-stage time synchronization and resampling, explicit marking is carried out on missing, noise and abnormal points, the weight of a subsequent algorithm can be reduced or skipped according to marks, and misjudgment caused by phase errors and dirty data is reduced.

The invention discloses a customer complaint work order automatic circulation system and method based on composite intention disassembly. The system receives and preprocesses the multi-mode customer complaint data and initializes a service circulation context; business intention identification and business key slot position extraction are carried out; disassembling the composite business intention, and constructing an ordered work order execution queue of a directed acyclic graph structure through cyclic dependence verification; evaluating the work order circulation action with the maximum expected cumulative income by using a reinforcement learning model; when business knowledge query is involved, compliance evidence fragments are retrieved, sorted and output; and generating a candidate reply and a business operation instruction by the fine-tuned large language model, outputting the candidate reply after business compliance verification, and calling an API (Application Program Interface) of an underlying business system to execute entity business handling operation in a cross-system manner. The problems of data interaction and state synchronization among multi-source heterogeneous systems are solved, the automatic execution efficiency of the composite work order at the bottom layer node is improved, and the system unauthorized and dirty data risks caused by an uncertain instruction are effectively avoided.

Adaptive system probe action to minimize input / output dirty data transfers is described. In one or more implementations, a system includes a processor, a memory configured to store data, and a cache configured to store a portion of the data stored in the memory for execution by the processor. The system also includes a cache coherence controller including a cache line history. The cache coherence controller is configured detect a direct memory access request from an input / output device. The direct memory access request is associated with an input / output operation involving the data. The cache coherence controller is further configured to identify a cache line associated with the direct memory access request, and, in response to the cache line history including a dirty data transfer record corresponding to the cache line, selectively send a probe to the cache based on a state of the cache line.

The invention relates to a data cleaning method and device and a storage medium. The data cleaning method comprises the steps of obtaining a to-be-cleaned data set; a preset generative adversarial network model is called to process the to-be-cleaned data set to obtain cleaned data, a generator in the generative adversarial network model is used for generating the cleaned data, and a discriminator in the generative adversarial network model is used for determining the probability that the cleaned data generated by the generator is target type data; the target type data is dirty data or clean data; in a generative adversarial network model training stage, the generator is used for generating second type data based on the first type data; in the training stage of the generative adversarial network model, the discriminator is used for distinguishing the second type of data and the first type of data so as to determine the second type of data as non-target type of data. According to the data cleaning method and device, the to-be-cleaned data is generated into the cleaned data through the generative adversarial network model.

The invention relates to a data synchronization method and system for intelligent dirty data detection and restoration based on DataX. According to the method, an intelligent dirty data processing engine is embedded between a Reader plug-in and a Writer plug-in of DataX, type matching, format verification, constraint conflict pre-detection and abnormal value recognition are completed online, and type conversion, format standardization, abnormal value replacement or flexible writing are carried out on dirty data according to a preset or self-adaptive strategy. And finally, the Writer executes insertion, updating, skipping or log recording only according to an instruction, and a traceable governance log is generated in the whole process. According to the method, the synchronization-cleaning capability is embedded into the DataX framework, so that the synchronization success rate and the data quality are remarkably improved, the external ETL dependence and the artificial script cost are reduced, and the method has the advantages of high intelligence, high throughput and flexible configuration.

The invention discloses an execution method and device for a cache global flushing instruction, terminal equipment and a medium, and relates to the technical field of electronics. The method comprises the following steps: monitoring a memory access instruction bus in response to a pre-write-back operation function; identifying all dirty data in the cache system under the condition that the global buffer instruction is monitored; determining a target cache line according to the life count value of the dirty data in each cache line of the cache system; and pre-writing back operation is performed on the target cache line according to the dirty data corresponding to the target cache line. According to the execution method, dirty data can be screened and written back in advance by detecting the global buffer instruction in the instruction stream in real time, so that the operation burden of a CPU in the execution stage is reduced, it is ensured that Cache cleaning and other calculation tasks are conducted in parallel, the overall write-back time is shortened, the utilization efficiency of system resources is improved, and the system performance is improved. Therefore, the delay problem of the flashing operation is obviously reduced, and the response speed and the performance consistency of the system are improved.

The invention relates to the technical field of data real-time stream processing, in particular to a method and system for achieving FlinkSql statement configuration through a user-defined ETL process, and the method comprises the following steps: operation of a job development page, graphical configuration of the ETL process, column configuration and FlinkSql basic configuration items, functions, where conditions, join and output configuration, and generation, preview and storage of FlinkSql statements. The method and system for realizing FlinkSql statement configuration through the user-defined ETL process have the beneficial effects that the use threshold is reduced through visual operation through the user-defined ETL process nodes; different tables of a plurality of data sources can be associated by using join nodes, so that the data availability is expanded; the data can be cleaned by using the function configuration node, the data standard is standardized, and dirty data is removed.

The invention discloses a data cleaning method for ship information infrastructure fault diagnosis, and belongs to the field of data processing. The method specifically relates to a preprocessing method of ship information infrastructure monitoring data and a data cleaning method for distinguishing fault data and dirty data. Through data preprocessing, a consistent and complete data set is constructed for subsequent data cleaning. At the moment, abnormal values in the data set are divided into interference data and abnormal equipment states, the abnormal equipment state values have important significance for subsequent fault diagnosis, and the interference data can influence model training. According to the method, a mode of combining FP-Growth and DBSCAN algorithms is adopted, and the characteristics that cabin equipment is complex in coupling and one fault often generates derivative alarms are utilized, so that equipment fault points and interference points are reasonably distinguished. Through the ship information infrastructure fault diagnosis-oriented data cleaning method, the problems of fuzzy data quality and low model precision caused by application of traditional fault diagnosis to ship information infrastructures can be solved.

The invention relates to the technical field of data processing, and discloses a store stockout compensation method and device, and the method comprises the steps: obtaining the daily sales information of a store in M historical days under the condition of stockout of the store, and the sales information is an influence factor on the sales quantity of the store; if the historical M days of the store include at least one day without stockout, determining K days with the highest similarity from the historical M days according to the similarity between the sales information of each day in the historical M days of the store and the sales information of the day when the store is stockout; according to the sales quantity and the sales volume trend index of each hour in at least one hour of each day in the K days, the compensation quantity of the store in the stockout period is obtained, and the sales volume trend index is obtained according to the sales quantity of the store in the non-stockout period and the sales quantity of the store in the same period in the historical M days. Therefore, dirty data generated by stockout can be restored, clean and effective data are provided for subsequent prediction, and the stability of a supply chain is improved.

The invention relates to a data cleaning system and method based on a blood relationship network, electronic equipment, a computer readable medium and a computer program product. The method comprises the steps that a blood relationship network of data is constructed based on a graph database, in the blood relationship network, nodes represent data tables or fields, and edges represent blood relationships among the nodes; determining a target node to be cleaned in the blood relationship network; obtaining all driving edges of the target node; judging the driving edges one by one through a preset attribute strategy and an expansion strategy; and when the driving edge does not meet a preset attribute strategy and an expansion strategy, deleting the driving edge to clean data in the blood relationship network. According to the method, dirty data can be automatically and accurately identified and deleted, data quality and analysis accuracy are improved, data traceability and compliance are enhanced, manual intervention and errors are reduced, and data governance efficiency and reliability are comprehensively improved.

The invention discloses a standardized cleaning method for multi-source alarm data access and a related device, and relates to the technical field of IT operation and maintenance in the financial industry, and the method comprises the steps: obtaining synchronization task parameters configured by a user, including a task execution plan, a processing class adaptive to multiple protocols, and an analysis and mapping rule, starting a task to receive multi-source alarm data, and obtaining a synchronization task; dirty data are screened through processing class matching and then analyzed into indexable objects, the indexable objects are mapped into data of a unified structure, formatting processing (including type conversion, null value filling and integrity and consistency verification) is carried out, and standardized cleaning is completed. According to the method, multi-protocol compatibility and multi-source data unified access are realized, scripts do not need to be customized, and technical barriers and maintenance cost are reduced; through data screening and standardization processing, the data quality and operation and maintenance efficiency are improved, quick response to key alarms is facilitated, and the continuity of financial core services is guaranteed.

The invention relates to the technical field of data processing, and discloses a quotation analysis model construction method and device, equipment and a medium, and the method comprises the steps: obtaining original full-amount data, and labeling the original full-amount data according to a preset structured format to obtain initial full-amount data; wherein the original total data represents complete data including historical quotation data and newly added quotation data on that day; cleaning dirty data based on the initial total data to obtain processed total data; wherein the dirty data cleaning comprises the step of carrying out outlier data screening by utilizing a trained instruction fine tuning model; determining open source data as unstructured data, and comprehensively processing the unstructured data and the processed total data to obtain target training data; and training a large language model by using the target training data to obtain a quotation analysis model. The method has the beneficial effects of standard structured text and real-time response.

The invention provides a robot training data analysis method and system based on deep learning. The method comprises the steps that operation parameters reflecting the work intensity of a robot are acquired; acquiring original sensor data acquired by the robot; deducing the physical state deviation of an end effector of the robot based on the operation parameters; according to the physical state deviation, adjusting the original sensor data to obtain adjusted sensor data; and optimizing a robot control strategy based on the adjusted sensor data. The method can effectively identify and quantify the sensor data error of the robot caused by the physical state deviation, and solves the problem that the model performance is reduced due to the fact that a traditional deep learning system learns in dirty data by correcting the original sensor data, thereby improving the accuracy and efficiency of robot control strategy optimization, and improving the user experience. And invalid adjustment caused by error attribution is avoided.

The invention belongs to the technical field of power dispatching automation, and particularly relates to a power pumped storage data storage method, device and equipment and a medium. In order to solve the problem of power pumped storage data storage, the method identifies type field codes from a message of to-be-stored power pumped storage data, and classifies the data based on the type field codes. After data classification, each type of data is subjected to rationality verification and data null value verification. The rationality verification ensures that the data accords with expected logic and business rules, and the data null value verification checks whether excessive null values or missing values exist in the data. Through data null value verification and rationality verification, low-quality data can be found and filtered in time, and the problem that a database is polluted by dirty data is avoided.

The invention discloses a laboratory data acquisition management method, system and device and a medium, and relates to the technical field of data acquisition management, and the method comprises the steps: associating data with a unique identifier of a sample to generate a standardized data packet with decision variables; the delay and retransmission rate are balanced through multi-objective optimization, and the batch write-in amount and the sub-library and sub-table strategy are dynamically adjusted; and marking abnormal data, classifying anomalies by adopting a clustering algorithm, and adaptively adjusting a detection threshold. According to the method, through dynamic partitioning of the message queues, adaptive adjustment of throughput is achieved, and data integrity is kept; the writing delay stability of the high-concurrency scene is kept through dynamic scaling of the batch writing amount; the query response time is shortened through an SSD / HDD sub-library strategy and time partition joint indexing; through a clustering algorithm, the anomaly classification accuracy is improved, and the false alarm rate is reduced; through transaction retry backoff, the data rollback accuracy is ensured when the database fails, and dirty data is prevented from being generated; and the network is optimized in real time through dynamic calculation of the network state.

Certain example embodiments relate to meta-learning based error detection. Base classifiers are provided for historical attributes in historical datasets. Each is trained to indicate dirtiness of a value for the associated historical attribute. Clusters and a clustering model are generated using historical clustering features determined for each historical attribute, which are then associated with the clusters. For each dirty attribute in a dirty dataset, corresponding dirty clustering features are determined. The dirty attributes are assigned to the clusters using the corresponding determined dirty clustering features and the clustering model. The base classifiers associated with the clusters to which the dirty attributes were assigned are retrieved. Dirty features are extracted from the dirty dataset, and selectively modified. The extracted dirty features are applied to the retrieved the base classifiers to determine meta-features. A meta-classifier is trained using labeled meta-features. Predictions about the dirty dataset's dirtiness can be made using the meta-classifier.

The invention discloses a data storage method, a data storage device and a medium, relates to the technical field of data storage, and provides a data storage method aiming at the write-in performance bottleneck caused by dirty data brushing under a cache pool when an existing distributed storage system stores data in a layered manner based on an erasure correction form. After to-be-written data is received, a data part with aligned stripes is directly written into a data pool. On one hand, data reading processing in the writing process is avoided, and the data writing speed of the data pool is increased. And on the other hand, the data pool is written by taking a group of stripes aligned with the stripes as a unit, which is equivalent to the realization of aggregation of the small block random IO, and the situation that the pressure of the small block random IO writing on the data pool becomes the bottleneck of the writing performance of the data pool is avoided. In conclusion, the data writing performance of the data pool when the data pool stores the data in the EC form is greatly improved, and the bottleneck problem of the writing performance of the data pool can be effectively solved.

This invention provides a word-level entity matching method for heterogeneous attributes, comprising: S1: obtaining a set of entity pairs and dividing the set into a training set and a test set; S2: constructing a cross-word matching matrix using the training set, and reconstructing word vectors using the cross-word matching matrix to obtain word-level matching vectors; S3: training a matching model using the word-level matching vectors to obtain a trained matching model; S4: matching the test set using the trained matching model to obtain entity matching results. This invention converts words in attributes into vectors, and constructs a cross-word matching matrix by comparing these vectors with the vectors of each word in the entity to be matched. The cross-word matching matrix contains the vector information of the entire entity, and can adaptively obtain suitable matching objects for each attribute. It has the advantages of high accuracy and robustness in entity matching between data, and can effectively handle the dirty data problem that occurs in entity matching.

The application relates to the field of data processing and provides a JSON data automatic structuring and warehousing method. The main idea is to solve the problem of how to store different JSON data structures of various data sources in a warehouse. The scheme includes judging the type of the accessed JSON data source, obtaining JSON data by using different methods according to the type, pre-processing the data, checking and processing dirty data to obtain standard JSON fields, parsing and processing the obtained JSON of different JSON data sources, agreeing on the format of the data file, generating a standard data file, investigating the structuring processing progress of the data, generating an ok file, checking the accuracy of the generated data file, obtaining the standard data file after the checking, and performing batch processing on the data file to store the data file in the warehouse.

The embodiment of the invention provides a dynamic phase modulation type heterogeneous data processing buffering device and method and a storage medium. The device comprises a data cursor, a data buffering pool, a dynamic phase modulation type data gate, a data processing pool, a data processor and a dirty data pool. The data buffer pool comprises N buffer data channels, and the dynamic phase modulation type data gates comprise N data gates.

The invention relates to a machine learning-based power grid operation data cleaning method and system. The method comprises the following steps of obtaining real-time operation data of a power grid structure; sequentially performing low-dimensional representation and data reconstruction on the real-time operation data by adopting a pre-trained auto-encoder model, and performing screening based on the reconstructed data to obtain abnormal data and normal data; based on the abnormal data, adopting an Apriori algorithm to classify the abnormal data into abnormal state data and dirty data, and removing the dirty data; and adopting a pre-trained GRU recurrent neural network model to complement the abnormal data from which the dirty data is removed to obtain the complemented abnormal data, and completing the data cleaning process. Compared with the prior art, the method has the advantages of realizing efficient and low-threshold power grid operation data cleaning and the like.

The invention discloses a phase modifier edge diagnosis system and method based on lightweight AI, relates to the technical field of power management, and aims to solve the technical problem of large automatic defect of current phase modifier diagnosis. Comprising a multi-mode sensing module, a lightweight AI edge deployment module, a network communication module, a data processing and storage module and an autonomous decision-making and fault diagnosis module, and the lightweight AI edge deployment module comprises an adaptive unit to avoid total model transmission. According to the system, significant technical gain is realized through the lightweight AI edge deployment module, the adaptive unit is embedded into the working condition feature mapping layer, model weight is dynamically adjusted in combination with working condition parameters such as the load rate of the phase modifier and the voltage of the power grid, misinformation caused by dirty data input is reduced, and the reliability of the system is improved. And the edge increment updating unit only transmits the deviation sample to perform local parameter updating, so that the diagnosis accuracy, the working condition adaptability and the operation efficiency are comprehensively improved, and efficient and reliable technical support is provided for the edge diagnosis of the phase modifier.

The present invention provides a data cleaning method and system based on a data security sandbox, and relates to the field of data cleaning. Technical solution: extract and process data in a data security sandbox to obtain original data, further process to obtain a feature vector set, and classify to obtain several vector groups; resample the original data and divide it into a training set and a test set, introduce a feature difference weighting mechanism to stack and combine multiple enhanced feature encoders to build an initial model, train and test the initial model based on the training set and the test set to obtain a SEFE model; use the SEFE model to denoise the feature vector set and then perform outlier detection to obtain outliers, and mark outliers that exceed the judgment standard as abnormal data; calculate the difference between the distribution of abnormal data and the distribution of normal data after clustering, rate the abnormal data with a difference exceeding a preset threshold as dirty data, and extract the dirty data for data management. The present invention improves the efficiency, speed and screening capabilities of data cleaning for a data security sandbox.

The invention relates to a smudginess regulation control method and control device for surface cleaning equipment, and the method comprises the steps: obtaining a plurality of preset monitoring time windows, collecting smudginess induction values of the surface cleaning equipment in a first monitoring time window, and storing the smudginess induction values to obtain a first smudginess data set; comparing data in the first smudginess data set with the obtained initial smudginess judgment threshold value to obtain the smudginess degree of the target area; adjusting the cleaning behavior according to the smudginess degree of the target area; based on the first smudginess data set, a first smudginess judgment threshold value is calculated through a preset statistical algorithm, the first smudginess judgment threshold value is used for replacing the initial smudginess judgment threshold value and is used for comparison of a next monitoring time window, and the steps are repeated. The method can solve the problem of systematic drifting of the reference reading of the sensor, and realizes the accuracy of smudginess detection and the reliability of control in the whole equipment life cycle.

The application discloses a makeup transfer model training method and a makeup transfer method based on cross-identity triplets, and relates to the technical field of computer vision and generative artificial intelligence. The model training method is divided into two stages of offline data construction and model training. In the offline data construction stage, a makeup prompt word dictionary is generated by a large language model; a homologous local rendering is performed on a nude face image cluster to generate a made-up image; a face mask is used to perform structure, background and rendering effectiveness filtering and semantic consistency verification, and a large number of high-quality cross-identity triplets are constructed through cross combination; a pre-trained diffusion model is fine-tuned under the strong supervision of a fixed text prompt and a double-path image condition, a conversion from text control to pure image control is realized, and a makeup transfer model is obtained. The application solves the problems of cross-identity makeup data scarcity, text semantic ambiguity and dirty data interference, realizes precise, lossless and what-you-see-is-what-you-get makeup transfer, and is suitable for film and television makeup, digital person rendering and e-commerce virtual makeup testing scenes.

This application provides a data update method, apparatus, computer device, and storage medium, relating to cloud technology, big data, cloud storage, and other technical fields. By determining an access queue based on an access index when at least two access requests trigger a data update, multiple access requests are accurately and conveniently centralized in the queue for unified management. The head request in the access queue updates the index data, and the head thread corresponding to the head request wakes up the waiting threads corresponding to the other access requests, allowing the remaining access requests to directly access the updated index data without repeating the update process. This avoids duplicate updates by a large number of access requests. Especially in high-concurrency scenarios, it prevents massive concurrent requests from penetrating the backend device, avoiding update errors, dirty data generation, operating system crashes, and other problems caused by simultaneous duplicate updates, ensuring the accuracy and reliability of the data update process.

The embodiment of the invention provides an operator testing method and device and a storage medium, and relates to the technical field of artificial intelligence chips, and the method comprises the steps: distributing a target video memory space for operator testing in an artificial intelligence chip, and filling the target video memory space with dirty data to simulate the residual data state of a video memory during the actual operation of the artificial intelligence chip; copying the test data to a target video memory space; reading an input tensor containing test data from the target video memory space, executing the to-be-measured sub-unit to obtain an output tensor, and writing the output tensor to the target video memory space; and reading the output tensor from the target video memory space, comparing the output tensor with the reference execution result to obtain a target test result, and carrying out operator test in a simulated chip real environment, so that memory operation defects in operator implementation can be forcibly exposed, meanwhile, the influence of the initial state of the video memory on operator function test can be found in time, and the test efficiency is improved. Therefore, the operator testing accuracy is improved.

The present invention discloses a data isolation method, device, equipment and storage medium for a stress testing scenario. The method includes: dividing and routing all traffic flowing into a system gateway to obtain processed traffic; identifying the processed traffic through middleware, wherein traffic with a stress testing identifier is stress testing traffic, and traffic without a stress testing identifier is real traffic; inputting the identified real traffic into the main database of the business, and inputting the data generated by the identified stress testing traffic into the main database in the in-memory database. The present invention adopts the method to achieve complete data isolation, will not generate dirty data, will not affect normal online users, can be used directly in a real online environment, obtain real performance indicator data, and provide an accurate and reliable reference for system expansion and operation and maintenance.