174 results about "Dirty data" patented technology

Certain example embodiments relate to dashboards that help streamline and automate data quality management processes used with machine learning (ML) models and ML-enabled technology. A clean dataset is initialized from a dirty dataset. A search space is the set of all possible combinations of available error detection algorithms and data repair algorithms. A scoring function measures performance of a given error detection algorithm and data repair algorithm combination on the clean dataset. An ML model is trained using the clean dataset. Best error detection and data repair algorithms are selected, based on an optimization on the set of all possible combinations, and the defined scoring function. The selected best error detection algorithm is applied to the clean dataset, and a repaired dataset is generated using the selected best repair algorithm. The clean dataset is set to the repaired dataset. This procedure is repeated until a condition is met.

The invention discloses a cache space control method of a storage system, electronic equipment and a storage medium, and relates to the technical field of data caching, and the method comprises the steps of identifying dirty data, comparing the size relation between the hit rate of cached data in a cache space and a first hit rate threshold value and a second hit rate threshold value, and when the hit rate is greater than the first hit rate threshold value, controlling the cache space of the storage system. When the hit rate is smaller than a first hit rate threshold value, executing an expansion instruction to control the cache module to expand a cache space, and when the hit rate is smaller than a second hit rate threshold value, controlling the cache module to brush dirty data down to a rear-end memory, executing a reduction instruction to control the cache module to reduce the cache space, and performing a cache hit rate judgment mechanism and a linkage telescopic cache mechanism in a full random access scene. The problems of performance reduction, high delay of data access, high power consumption and the like of the cache system due to the fact that the cache space is fixed and cannot adapt to different application scenes and changes of data loads in the prior art are solved, performance self-adaption of the IO model is achieved, the response speed is increased, and the performance of the cache system in multiple storage scenes is improved.

The invention relates to the technical field of artificial intelligence, in particular to an electrical connection safety management and control method based on intelligent monitoring, which comprises the steps of data acquisition, time synchronization, quality evaluation, feature engineering and the like. According to the method, electric, thermal, mechanical, acoustic and other channels are placed on the same time axis through unified alignment and quality marking of multi-modal data and two-stage time synchronization and resampling, explicit marking is carried out on missing, noise and abnormal points, the weight of a subsequent algorithm can be reduced or skipped according to marks, and misjudgment caused by phase errors and dirty data is reduced.

The invention discloses a customer complaint work order automatic circulation system and method based on composite intention disassembly. The system receives and preprocesses the multi-mode customer complaint data and initializes a service circulation context; business intention identification and business key slot position extraction are carried out; disassembling the composite business intention, and constructing an ordered work order execution queue of a directed acyclic graph structure through cyclic dependence verification; evaluating the work order circulation action with the maximum expected cumulative income by using a reinforcement learning model; when business knowledge query is involved, compliance evidence fragments are retrieved, sorted and output; and generating a candidate reply and a business operation instruction by the fine-tuned large language model, outputting the candidate reply after business compliance verification, and calling an API (Application Program Interface) of an underlying business system to execute entity business handling operation in a cross-system manner. The problems of data interaction and state synchronization among multi-source heterogeneous systems are solved, the automatic execution efficiency of the composite work order at the bottom layer node is improved, and the system unauthorized and dirty data risks caused by an uncertain instruction are effectively avoided.

Adaptive system probe action to minimize input / output dirty data transfers is described. In one or more implementations, a system includes a processor, a memory configured to store data, and a cache configured to store a portion of the data stored in the memory for execution by the processor. The system also includes a cache coherence controller including a cache line history. The cache coherence controller is configured detect a direct memory access request from an input / output device. The direct memory access request is associated with an input / output operation involving the data. The cache coherence controller is further configured to identify a cache line associated with the direct memory access request, and, in response to the cache line history including a dirty data transfer record corresponding to the cache line, selectively send a probe to the cache based on a state of the cache line.

The invention relates to a data cleaning method and device and a storage medium. The data cleaning method comprises the steps of obtaining a to-be-cleaned data set; a preset generative adversarial network model is called to process the to-be-cleaned data set to obtain cleaned data, a generator in the generative adversarial network model is used for generating the cleaned data, and a discriminator in the generative adversarial network model is used for determining the probability that the cleaned data generated by the generator is target type data; the target type data is dirty data or clean data; in a generative adversarial network model training stage, the generator is used for generating second type data based on the first type data; in the training stage of the generative adversarial network model, the discriminator is used for distinguishing the second type of data and the first type of data so as to determine the second type of data as non-target type of data. According to the data cleaning method and device, the to-be-cleaned data is generated into the cleaned data through the generative adversarial network model.

The invention relates to a data synchronization method and system for intelligent dirty data detection and restoration based on DataX. According to the method, an intelligent dirty data processing engine is embedded between a Reader plug-in and a Writer plug-in of DataX, type matching, format verification, constraint conflict pre-detection and abnormal value recognition are completed online, and type conversion, format standardization, abnormal value replacement or flexible writing are carried out on dirty data according to a preset or self-adaptive strategy. And finally, the Writer executes insertion, updating, skipping or log recording only according to an instruction, and a traceable governance log is generated in the whole process. According to the method, the synchronization-cleaning capability is embedded into the DataX framework, so that the synchronization success rate and the data quality are remarkably improved, the external ETL dependence and the artificial script cost are reduced, and the method has the advantages of high intelligence, high throughput and flexible configuration.

The invention discloses a data cleaning method for ship information infrastructure fault diagnosis, and belongs to the field of data processing. The method specifically relates to a preprocessing method of ship information infrastructure monitoring data and a data cleaning method for distinguishing fault data and dirty data. Through data preprocessing, a consistent and complete data set is constructed for subsequent data cleaning. At the moment, abnormal values in the data set are divided into interference data and abnormal equipment states, the abnormal equipment state values have important significance for subsequent fault diagnosis, and the interference data can influence model training. According to the method, a mode of combining FP-Growth and DBSCAN algorithms is adopted, and the characteristics that cabin equipment is complex in coupling and one fault often generates derivative alarms are utilized, so that equipment fault points and interference points are reasonably distinguished. Through the ship information infrastructure fault diagnosis-oriented data cleaning method, the problems of fuzzy data quality and low model precision caused by application of traditional fault diagnosis to ship information infrastructures can be solved.

The invention relates to the technical field of data processing, and discloses a store stockout compensation method and device, and the method comprises the steps: obtaining the daily sales information of a store in M historical days under the condition of stockout of the store, and the sales information is an influence factor on the sales quantity of the store; if the historical M days of the store include at least one day without stockout, determining K days with the highest similarity from the historical M days according to the similarity between the sales information of each day in the historical M days of the store and the sales information of the day when the store is stockout; according to the sales quantity and the sales volume trend index of each hour in at least one hour of each day in the K days, the compensation quantity of the store in the stockout period is obtained, and the sales volume trend index is obtained according to the sales quantity of the store in the non-stockout period and the sales quantity of the store in the same period in the historical M days. Therefore, dirty data generated by stockout can be restored, clean and effective data are provided for subsequent prediction, and the stability of a supply chain is improved.

The invention relates to a data cleaning system and method based on a blood relationship network, electronic equipment, a computer readable medium and a computer program product. The method comprises the steps that a blood relationship network of data is constructed based on a graph database, in the blood relationship network, nodes represent data tables or fields, and edges represent blood relationships among the nodes; determining a target node to be cleaned in the blood relationship network; obtaining all driving edges of the target node; judging the driving edges one by one through a preset attribute strategy and an expansion strategy; and when the driving edge does not meet a preset attribute strategy and an expansion strategy, deleting the driving edge to clean data in the blood relationship network. According to the method, dirty data can be automatically and accurately identified and deleted, data quality and analysis accuracy are improved, data traceability and compliance are enhanced, manual intervention and errors are reduced, and data governance efficiency and reliability are comprehensively improved.

The invention discloses a standardized cleaning method for multi-source alarm data access and a related device, and relates to the technical field of IT operation and maintenance in the financial industry, and the method comprises the steps: obtaining synchronization task parameters configured by a user, including a task execution plan, a processing class adaptive to multiple protocols, and an analysis and mapping rule, starting a task to receive multi-source alarm data, and obtaining a synchronization task; dirty data are screened through processing class matching and then analyzed into indexable objects, the indexable objects are mapped into data of a unified structure, formatting processing (including type conversion, null value filling and integrity and consistency verification) is carried out, and standardized cleaning is completed. According to the method, multi-protocol compatibility and multi-source data unified access are realized, scripts do not need to be customized, and technical barriers and maintenance cost are reduced; through data screening and standardization processing, the data quality and operation and maintenance efficiency are improved, quick response to key alarms is facilitated, and the continuity of financial core services is guaranteed.

The invention provides a robot training data analysis method and system based on deep learning. The method comprises the steps that operation parameters reflecting the work intensity of a robot are acquired; acquiring original sensor data acquired by the robot; deducing the physical state deviation of an end effector of the robot based on the operation parameters; according to the physical state deviation, adjusting the original sensor data to obtain adjusted sensor data; and optimizing a robot control strategy based on the adjusted sensor data. The method can effectively identify and quantify the sensor data error of the robot caused by the physical state deviation, and solves the problem that the model performance is reduced due to the fact that a traditional deep learning system learns in dirty data by correcting the original sensor data, thereby improving the accuracy and efficiency of robot control strategy optimization, and improving the user experience. And invalid adjustment caused by error attribution is avoided.

The invention discloses a laboratory data acquisition management method, system and device and a medium, and relates to the technical field of data acquisition management, and the method comprises the steps: associating data with a unique identifier of a sample to generate a standardized data packet with decision variables; the delay and retransmission rate are balanced through multi-objective optimization, and the batch write-in amount and the sub-library and sub-table strategy are dynamically adjusted; and marking abnormal data, classifying anomalies by adopting a clustering algorithm, and adaptively adjusting a detection threshold. According to the method, through dynamic partitioning of the message queues, adaptive adjustment of throughput is achieved, and data integrity is kept; the writing delay stability of the high-concurrency scene is kept through dynamic scaling of the batch writing amount; the query response time is shortened through an SSD / HDD sub-library strategy and time partition joint indexing; through a clustering algorithm, the anomaly classification accuracy is improved, and the false alarm rate is reduced; through transaction retry backoff, the data rollback accuracy is ensured when the database fails, and dirty data is prevented from being generated; and the network is optimized in real time through dynamic calculation of the network state.

Certain example embodiments relate to meta-learning based error detection. Base classifiers are provided for historical attributes in historical datasets. Each is trained to indicate dirtiness of a value for the associated historical attribute. Clusters and a clustering model are generated using historical clustering features determined for each historical attribute, which are then associated with the clusters. For each dirty attribute in a dirty dataset, corresponding dirty clustering features are determined. The dirty attributes are assigned to the clusters using the corresponding determined dirty clustering features and the clustering model. The base classifiers associated with the clusters to which the dirty attributes were assigned are retrieved. Dirty features are extracted from the dirty dataset, and selectively modified. The extracted dirty features are applied to the retrieved the base classifiers to determine meta-features. A meta-classifier is trained using labeled meta-features. Predictions about the dirty dataset's dirtiness can be made using the meta-classifier.

This invention provides a word-level entity matching method for heterogeneous attributes, comprising: S1: obtaining a set of entity pairs and dividing the set into a training set and a test set; S2: constructing a cross-word matching matrix using the training set, and reconstructing word vectors using the cross-word matching matrix to obtain word-level matching vectors; S3: training a matching model using the word-level matching vectors to obtain a trained matching model; S4: matching the test set using the trained matching model to obtain entity matching results. This invention converts words in attributes into vectors, and constructs a cross-word matching matrix by comparing these vectors with the vectors of each word in the entity to be matched. The cross-word matching matrix contains the vector information of the entire entity, and can adaptively obtain suitable matching objects for each attribute. It has the advantages of high accuracy and robustness in entity matching between data, and can effectively handle the dirty data problem that occurs in entity matching.

The application relates to the field of data processing and provides a JSON data automatic structuring and warehousing method. The main idea is to solve the problem of how to store different JSON data structures of various data sources in a warehouse. The scheme includes judging the type of the accessed JSON data source, obtaining JSON data by using different methods according to the type, pre-processing the data, checking and processing dirty data to obtain standard JSON fields, parsing and processing the obtained JSON of different JSON data sources, agreeing on the format of the data file, generating a standard data file, investigating the structuring processing progress of the data, generating an ok file, checking the accuracy of the generated data file, obtaining the standard data file after the checking, and performing batch processing on the data file to store the data file in the warehouse.

The embodiment of the invention provides a dynamic phase modulation type heterogeneous data processing buffering device and method and a storage medium. The device comprises a data cursor, a data buffering pool, a dynamic phase modulation type data gate, a data processing pool, a data processor and a dirty data pool. The data buffer pool comprises N buffer data channels, and the dynamic phase modulation type data gates comprise N data gates.

The invention discloses a phase modifier edge diagnosis system and method based on lightweight AI, relates to the technical field of power management, and aims to solve the technical problem of large automatic defect of current phase modifier diagnosis. Comprising a multi-mode sensing module, a lightweight AI edge deployment module, a network communication module, a data processing and storage module and an autonomous decision-making and fault diagnosis module, and the lightweight AI edge deployment module comprises an adaptive unit to avoid total model transmission. According to the system, significant technical gain is realized through the lightweight AI edge deployment module, the adaptive unit is embedded into the working condition feature mapping layer, model weight is dynamically adjusted in combination with working condition parameters such as the load rate of the phase modifier and the voltage of the power grid, misinformation caused by dirty data input is reduced, and the reliability of the system is improved. And the edge increment updating unit only transmits the deviation sample to perform local parameter updating, so that the diagnosis accuracy, the working condition adaptability and the operation efficiency are comprehensively improved, and efficient and reliable technical support is provided for the edge diagnosis of the phase modifier.

The invention relates to a smudginess regulation control method and control device for surface cleaning equipment, and the method comprises the steps: obtaining a plurality of preset monitoring time windows, collecting smudginess induction values of the surface cleaning equipment in a first monitoring time window, and storing the smudginess induction values to obtain a first smudginess data set; comparing data in the first smudginess data set with the obtained initial smudginess judgment threshold value to obtain the smudginess degree of the target area; adjusting the cleaning behavior according to the smudginess degree of the target area; based on the first smudginess data set, a first smudginess judgment threshold value is calculated through a preset statistical algorithm, the first smudginess judgment threshold value is used for replacing the initial smudginess judgment threshold value and is used for comparison of a next monitoring time window, and the steps are repeated. The method can solve the problem of systematic drifting of the reference reading of the sensor, and realizes the accuracy of smudginess detection and the reliability of control in the whole equipment life cycle.

The application discloses a makeup transfer model training method and a makeup transfer method based on cross-identity triplets, and relates to the technical field of computer vision and generative artificial intelligence. The model training method is divided into two stages of offline data construction and model training. In the offline data construction stage, a makeup prompt word dictionary is generated by a large language model; a homologous local rendering is performed on a nude face image cluster to generate a made-up image; a face mask is used to perform structure, background and rendering effectiveness filtering and semantic consistency verification, and a large number of high-quality cross-identity triplets are constructed through cross combination; a pre-trained diffusion model is fine-tuned under the strong supervision of a fixed text prompt and a double-path image condition, a conversion from text control to pure image control is realized, and a makeup transfer model is obtained. The application solves the problems of cross-identity makeup data scarcity, text semantic ambiguity and dirty data interference, realizes precise, lossless and what-you-see-is-what-you-get makeup transfer, and is suitable for film and television makeup, digital person rendering and e-commerce virtual makeup testing scenes.

This application provides a data update method, apparatus, computer device, and storage medium, relating to cloud technology, big data, cloud storage, and other technical fields. By determining an access queue based on an access index when at least two access requests trigger a data update, multiple access requests are accurately and conveniently centralized in the queue for unified management. The head request in the access queue updates the index data, and the head thread corresponding to the head request wakes up the waiting threads corresponding to the other access requests, allowing the remaining access requests to directly access the updated index data without repeating the update process. This avoids duplicate updates by a large number of access requests. Especially in high-concurrency scenarios, it prevents massive concurrent requests from penetrating the backend device, avoiding update errors, dirty data generation, operating system crashes, and other problems caused by simultaneous duplicate updates, ensuring the accuracy and reliability of the data update process.

The embodiment of the invention provides an operator testing method and device and a storage medium, and relates to the technical field of artificial intelligence chips, and the method comprises the steps: distributing a target video memory space for operator testing in an artificial intelligence chip, and filling the target video memory space with dirty data to simulate the residual data state of a video memory during the actual operation of the artificial intelligence chip; copying the test data to a target video memory space; reading an input tensor containing test data from the target video memory space, executing the to-be-measured sub-unit to obtain an output tensor, and writing the output tensor to the target video memory space; and reading the output tensor from the target video memory space, comparing the output tensor with the reference execution result to obtain a target test result, and carrying out operator test in a simulated chip real environment, so that memory operation defects in operator implementation can be forcibly exposed, meanwhile, the influence of the initial state of the video memory on operator function test can be found in time, and the test efficiency is improved. Therefore, the operator testing accuracy is improved.

The present invention discloses a data isolation method, device, equipment and storage medium for a stress testing scenario. The method includes: dividing and routing all traffic flowing into a system gateway to obtain processed traffic; identifying the processed traffic through middleware, wherein traffic with a stress testing identifier is stress testing traffic, and traffic without a stress testing identifier is real traffic; inputting the identified real traffic into the main database of the business, and inputting the data generated by the identified stress testing traffic into the main database in the in-memory database. The present invention adopts the method to achieve complete data isolation, will not generate dirty data, will not affect normal online users, can be used directly in a real online environment, obtain real performance indicator data, and provide an accurate and reliable reference for system expansion and operation and maintenance.

The application discloses a data writing method, device, equipment and medium, and relates to the technical field of data transmission. The method is applied to a RAID provided with a DDR and comprises the following steps: acquiring and marking data in the DDR as dirty data and historical data in a dirty RAM, and transmitting a write operation to an AXI slave; when the write operation hits a tag, the historical data covered by the dirty data is discarded according to a dirty identifier in the dirty RAM, and is used to replace the historical data of a certain CacheLine in the Cache; the existing data writing method spends more time to determine whether the data in the CacheLine should be kept or covered, when the written data has more segments, the CacheLine is filled to perform multiple discontinuous read operations, and great performance loss is caused. The data processing of the hit tag is determined, the dirty identifier avoids unnecessary write-back operations, and the write-in performance of the data writing is improved.

This invention discloses a method and system for processing the entire lifecycle of procurement contracts based on a unified stored procedure with multiple TVPs and AI protection verification, belonging to the field of industrial software MES / ERP technology. This invention uses three sets of table-valued parameters (TVPs) as a unified data entry point, integrating six core operations—adding, modifying, deleting, appending details, modifying details, and deleting details—in a single stored procedure, and returning unified exception information through a single output parameter. The system incorporates a two-level centralized parameter verification mechanism. The first level performs a one-time empty table check on the three sets of TVP tables; if any table is empty, the transaction is immediately rolled back and an exception is returned. The second level performs centralized integrity and legality verification on all business parameters, achieving security protection for AI and external calls, effectively intercepting dirty data and illegal requests. This invention features a simple architecture, high execution efficiency, and comprehensive business coverage. The transaction mechanism ensures strong consistency of master-slave table data, supports compatible operation with both SQL Server and GaussDB databases, and can securely integrate into the AI ​​ecosystem without business restructuring. It is particularly suitable for the high-stability and high-security procurement contract business processing of complex factories with more than 11 production lines. This invention has been fully implemented in a real industrial ERP system and can run stably in a real production environment for a long time, demonstrating mature engineering value and industrial promotion capabilities.

Embodiments of this disclosure provide a cache, a control method therefor, and a computer system, and relate to the field of storage technologies, to improve memory access efficiency of the computer system. The cache is connected to a memory controller, and the cache includes a plurality of cache lines. The control method for a cache includes: storing write data of a received write command into cache lines, and before the cache lines are allocated to new memory addresses, sending dirty data stored in the cache lines to the memory controller.

The embodiment of the present application provides a kind of data cleaning method and device, the method includes determining data attribute field from each business field of business scene, by data attribute field, determine the search field set from each database table, and determine the database table set matched with search field set, the source data associated with business scene is analyzed, determine the business data value of the business field to be cleaned, according to search field set and database table set, the structured query language sql data cleaning script corresponding to the business data value of the business field to be cleaned is constructed, and the relevant business data in the database table set associated with the business data value of the business field to be cleaned is cleaned by sql data cleaning script.It is thus, the scheme can realize the accurate cleaning of dirty data automatically by the constructed sql data cleaning script, so as to effectively reduce the time consumed by test personnel for cleaning dirty data.

A cache controller of a cache assigns a dirty tracking bit for each dirty byte of a cache line. Once a predetermined interval has elapsed without any accesses to the cache line or to a cache set that includes the cache line, the cache controller compresses contiguous dirty tracking bits for each portion of the cache line. Compressing the dirty tracking bits for contiguous dirty portions of the cache line allows the cache to store more dirty data using fewer dirty tracking bits, reducing area cost and bandwidth among levels of a memory hierarchy.

The application discloses a data storage method, comprising the following steps: in response to receiving a write IO, obtaining a current cache strategy; in response to the current cache strategy being a first cache strategy, judging whether there is unexpired dirty data in the current cache; in response to there being unexpired dirty data, copying the write IO to obtain two write IOs and judging whether the write IO hits the unexpired dirty data; in response to hitting the unexpired dirty data, directly writing one of the write IOs into a RAID and updating the hit unexpired dirty data by the other write IO; and in response to one of the write IOs being successfully written into the RAID, performing invalidation processing on the hit unexpired dirty data. The application further discloses a system, a computer device and a readable storage medium. When the cache strategy is switched from WRITE-BACK to WRITE-THROUGH, the scheme provided by the application does not cause permanent data loss.