89 results about "Data sanitization" patented technology

The invention relates to the technical field of intelligent decision making, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a decision making method, device, equipment and medium based on multi-modal data, comprising: collecting and cleaning multi-modal original data, executing format conversion and desensitization processing, and generating standardized data; multi-modal features are extracted and fused based on the standardized data, and multi-modal feature vectors are generated; performing reasoning on the multi-modal feature vector through a reasoning model to generate reasoning result data; pushing the reasoning result data to a review terminal, and receiving correction feedback data; and updating parameters of the reasoning model based on the corrected feedback data, and generating an updated reasoning model. According to the method, after data cleaning, format conversion and desensitization processing, multi-modal feature extraction and reasoning model reasoning processing are executed, effective fusion and accurate reasoning of data multi-modal features are achieved, and decision accuracy and data safety are improved.

The invention discloses a multi-scene application data acquisition method based on atomization design. The method comprises the following steps: analyzing reported data into an atomization data structure; performing semantic recognition on the data labels to generate a label semantic mapping result; constructing a layered cache structure; generating a compressed semantic mapping rule by applying a multi-level semantic difference coding algorithm; identifying an optimal query path from the tag affinity matrix; and executing historical data cleaning based on the data value. Intelligent mapping of data labels is achieved through semantic recognition and version control, the storage efficiency is improved through predictive lazy loading and compression coding, the query path is optimized based on the affinity matrix, and the technical problem of multi-scene application data collection is effectively solved.

Systems and methods for intelligent data sanitization employing PCM and AI / ML are provided. The idea uses AI / ML to detect specific facts that needs sanitization rather than full properties in incoming records. Data sanitization is optimized using this focused method, saving computational resources. To properly manage changing data volumes, PCM shifts between Logical 0 and Logical 1 states. Logical 0 processes smaller volumes with high resistance and low conductivity, while Logical 1 processes large volumes with low resistance and high conductivity. The AI / ML module organizes and directs data to maximize resource and processing efficiency. The PCM processes data in-memory and directly overwrites, eliminating erasure. AI / ML and PCM integrate to sanitize data quickly, efficiently, and securely, improving system performance and data integrity without a central repository. The system dynamically adjusts to changing data patterns, protecting and optimizing data.

The invention provides a building structure intelligent design method and system based on 3D modeling, electronic equipment and a storage medium, and relates to the field of building intelligent design, and the method comprises the following steps: obtaining three-dimensional data through an unmanned aerial vehicle, oblique photography and laser scanning, and carrying out data cleaning, conversion and screening; performing feature extraction and semantic segmentation on the point cloud data by using a deep learning model; performing intelligent registration and fusion on the two types of point clouds by adopting a deep learning model; a three-dimensional model is generated based on the registration data, a parameter form is generated in combination with the building planning data, and a component model is constructed; collision detection, smooth verification and mechanical simulation are carried out on the whole model, parameter adjustment and optimization are carried out according to verification results, through automatic data collection, multi-source point cloud fusion, intelligent parameter modeling and multi-dimensional verification and optimization, the problems that a traditional method is low in efficiency and insufficient in precision are solved, and the design efficiency and precision are remarkably improved.

The invention provides a large-scale feature data storage and query method. The large-scale feature data storage and query method comprises the following steps: S1) feature data storage: writing real-time feature data into an original column cluster of HBase and retaining a preset number of historical versions; s2) regularly merging historical data: regularly merging historical version data in the original column cluster through a batch processing task, and writing the merged data into a new summary column cluster; s3) feature data query: when feature data is queried, preferentially reading data from the summarized column cluster, and if the data is not found, returning to the original column cluster for query; and S4) dynamic scheduling and data cleaning: dynamically adjusting a feature data merging strategy according to the access frequency of the feature data, and regularly cleaning out-of-date data.

The invention relates to the technical field of system database management, in particular to an intelligent monitoring analysis method and system for gate opening and closing working condition data, and the method comprises the steps: obtaining fault records from a historical fault data storage library of a specified gate, and constructing a fault feature library; acquiring historical monitoring data of the sensor group, performing pattern recognition and statistical analysis to form a fault feature weight table, and storing the fault feature weight table in a data cleaning database; and classifying the sensor group through a database index technology by utilizing the fault feature weight table, the external prediction data and the sensor installation position data, and storing a classification result into a sensor level index table. And a distributed data intelligent monitoring system is formed through N independent data processing units in combination with a database interface protocol. The data processing unit comprises a data acquisition layer, a data real-time analysis layer, a data cleaning and quality control layer and a data storage and retrieval layer, and high-efficiency processing and retrieval of data are ensured.

Embodiments of the present disclosure may provide a data clean room allowing secure data analysis across multiple accounts, without the use of third parties. Each account may be associated with a different company or party. The data clean room may provide security functions to safeguard sensitive information. For example, the data clean room may restrict access to data in other accounts. The data clean room may also restrict which data may be used in the analysis and may restrict the output. The overlap data may be anonymized to prevent sensitive information from being revealed.

The invention relates to the technical field of thermal power plant abnormal data processing, in particular to a thermal power plant production monitoring system network security data set abnormal data identification and cleaning method, which comprises the following steps: mapping a spatial-temporal feature vector to a preset thermal power plant multi-level causal graph to obtain a data causal graph; performing information propagation and node updating on the data causal graph, and calculating an abnormal score of each node in a preset thermal power plant multilevel causal graph; identifying abnormal data according to the abnormal score and the network security data set; and cleaning the abnormal data according to the abnormal type and a preset data cleaning strategy. According to the method, correlation modeling of data on time and a system topological structure is realized through a multi-level cause and effect graph; an abnormal score is calculated through information spreading and node updating, and an abnormal event in the network security data can be accurately identified; the abnormal data is cleaned based on the preset data cleaning strategy, redundant, wrong and abnormal data can be effectively removed, and meanwhile key risk information is reserved.

Methods and systems are described herein for leveraging artificial intelligence to sanitize sensitive data and prevent the data from leaving the mobile device and / or be exposed to unauthorized third parties. More specifically, methods and systems are described for a novel and unconventional architecture for a data sanitization application, a novel and unconventional delivery format for the data sanitization model, and a novel and unconventional output format of the data sanitization model.

A system and method for automated bulk creation of Universal Resource Locators (URLs) with embedded Urchin Tracking Module (UTM) parameters is disclosed. The method processes input data, including endpoint URLs and UTM parameters, through pre-processing steps such as data cleansing and format standardization. The method appends UTM parameter-value pairs to endpoint URLs, optionally encoding the parameters. The generated tracking URLs are stored in a structured database for seamless integration with analytics tools, enabling reliable tracking and actionable insights into web traffic and user behavior.

The invention discloses a packet management method and device, equipment and a storage medium, and the method comprises the steps: obtaining a processing instruction for a target packet, and publishing the target packet in a version control system Git; executing the processing instruction to perform at least one of installation, data cleaning and removal on the target packet from the Git; and monitoring the processed target packet to obtain a monitoring result and outputting the monitoring result. Therefore, the packet management scheme provided by the embodiment of the invention can be deeply integrated with the Git, so that packet management can be realized by virtue of a private packet management function supported by the Git without depending on a public network packet manager, thereby reducing the risk of information leakage and improving the information security. Moreover, packet management is realized by means of Git, and additional privatized packet management does not need to be established or additional server hosting does not need to be configured, so that the cost can be reduced.

A system for the near real-time exchange of regulatory-compliant and privacy-compliant consumer credit data between two parties (i.e., a credit bureau and a financial services institution) that facilitates the financial services institutions (i.e., lenders) to directly identify audiences of pre-screened consumers (i.e., prospects) to offer credit products without requiring the intervention of a third party, such as an agent of the bureau, employs a data clean room to facilitate the necessary data exchange. The clean room enables and expedites the presentment of firm offer of credit pre-screened offers across authenticated or authenticatable offline and digital online channels, thereby making credit available to consumers in a more timely and efficient manner while protecting consumer privacy.

The invention discloses a data processing method and device, equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: judging whether a current service table meets a preset table rotation condition or not based on the service table information of the current service table; under the condition that the current service table meets a preset table rotation condition, judging whether data cleaning of the data in the empty table is completed or not; and under the condition that data cleaning of the data in the vacant table is completed, taking the vacant table as a new current service table to respond to the service request. According to the invention, under the condition that the current service table meets the preset table rotation condition and the data in the vacant table is cleaned, the vacant table is used as the new current service table to respond to the service request. Compared with an existing mode of cleaning the data table in the service low-peak period, the method has the advantage that the table data can be cleaned under the condition that the online service is not paused by taking the cleaned empty table as the new current service table to respond to the service request.

The invention provides a prevention and control method and device for network-related illegal behaviors and a storage medium. The prevention and control method comprises the steps of obtaining network-related risk data; performing data cleaning, data conversion and feature selection on the network related risk data to obtain preprocessed multi-source heterogeneous data; inputting the multi-source heterogeneous data into a network-related susceptible population classification model, and determining a network-related susceptible population big data portrait; wherein the network-related susceptible population classification model is constructed based on a random forest algorithm; determining the network-related susceptible crowd according to the network-related susceptible crowd big data portrait; and the risk information of the network-related susceptible crowd is provided for related institutions. According to the method, deep acquisition, fine analysis and scientific modeling are carried out on the multi-source heterogeneous data, accurate identification and early warning of novel network-related susceptible crowds are realized, effective clues and laws hidden behind complex data are effectively identified, and the working efficiency of illegal risk disposal is improved.

The application discloses a systematic color SVG icon dataset construction method and device, and the method comprises the following steps: collecting SVG files, and performing data cleaning on the collected SVG files to obtain initial files; wherein the SVG files comprise SVG icons of different categories, and the initial files are color SVG files; performing standardization processing on the boundary box size of the initial files to obtain sample files; classifying the sample files, and dividing training set, verification set and test set according to the classified sample files to obtain SVG datasets. The application can improve the construction quality of color SVG datasets.

Systems and methods for intelligent data sanitization employing PCM and AI / ML are provided. The idea uses AI / ML to detect specific facts that needs sanitization rather than full properties in incoming records. Data sanitization is optimized using this focused method, saving computational resources. To properly manage changing data volumes, PCM shifts between Logical 0 and Logical 1 states. Logical 0 processes smaller volumes with high resistance and low conductivity, while Logical 1 processes large volumes with low resistance and high conductivity. The AI / ML module organizes and directs data to maximize resource and processing efficiency. The PCM processes data in-memory and directly overwrites, eliminating erasure. AI / ML and PCM integrate to sanitize data quickly, efficiently, and securely, improving system performance and data integrity without a central repository. The system dynamically adjusts to changing data patterns, protecting and optimizing data.

There is provided a data preparation function in a wireless communication network, the data preparation function comprising: one or more processors arranged to: collect data from one or more data sources in the wireless communication network; analyse the collected data to derive one or more data characteristics and to identify whether the collected data face one or more quality issues or irregularities; and prepare the collected data based on the analysis, including performing one or more of the following: data recovery to recover data missing from the collected data; data cleaning of the collected data; formatting of the collected data; or separation of the collected data into different data sets for one or more training tasks.

The embodiment of the present application provides a kind of data cleaning method and device, the method includes determining data attribute field from each business field of business scene, by data attribute field, determine the search field set from each database table, and determine the database table set matched with search field set, the source data associated with business scene is analyzed, determine the business data value of the business field to be cleaned, according to search field set and database table set, the structured query language sql data cleaning script corresponding to the business data value of the business field to be cleaned is constructed, and the relevant business data in the database table set associated with the business data value of the business field to be cleaned is cleaned by sql data cleaning script.It is thus, the scheme can realize the accurate cleaning of dirty data automatically by the constructed sql data cleaning script, so as to effectively reduce the time consumed by test personnel for cleaning dirty data.

The present invention relates to a computer implemented method for data cleansing. In the method, a data pre-processing module of a first computing instance (3) obtains an at least two-dimensional un-shuffled data structure (10). The data structure (10) comprises healthcare patient data. The data structure (10) comprises data on a plurality of patient records. Each patient record comprises one or more fields wherein each field has a field type and wherein the patient records have a pre-shuffled order. The data pre-processing module of the first computing device shuffles fields of a first field type by moving fields having the first field types from one patient record to another patient record. Fields with the first field type have a post-shuffled order after the shuffling thereby creating a shuffled data structure(lO). A communications module transmits (5) the shuffled data structure (10) comprising the healthcare patient data over a communications network (7) from the first computing instance to a second computing instance.

We provide the system. [Solution] The information processing device is a means of collecting diverse data related to multiple businesses, A means of cleaning up and standardizing inaccurate data using collected data, A means for training and updating generative artificial intelligence models based on cleaned and standardized data, A means of automatically determining whether a business can continue using a trained model, A means for generating and reporting a report that includes the judgment results and recommendations, A system that includes this.

The present invention discloses a method, apparatus, and computer storage medium for authenticating user identities. Among them, the method includes: receiving an access request and extracting information to be verified from the access request; if there is a preset built-in topic in the distributed stream processing platform currently, determining whether the data in the preset built-in topic has been updated; the preset built-in topic is used to store authenticated user information, and the data cleaning policy of the preset built-in topic is a compression policy; if the data in the preset built-in topic has not been updated, obtaining the authenticated user information from the local cache to verify the information to be verified based on the obtained authenticated user information. The technical solution provided by the present invention can perform dynamic management of user information without restarting Kafka, thereby improving the real-time performance and security of authentication.

The present invention relates to a non - management - type database combination cleaning method for users. By querying the storage time of historical information in the database and arranging it in order; obtaining the data cleaning time range selected by the user and performing data cleaning on the historical information; constructing an automatic data cleaning process and detecting whether automatic data cleaning is required, the database combination cleaning process is realized. The present invention can avoid the one - size - fits - all cleaning method, authorize the user to have the right to choose. When the hard disk space is limited, the user can freely select the time range of the data to be deleted, and the equipment software is used to implement the data deletion operation, so that the utilization rate of the equipment hard disk is always controlled below the trigger threshold for automatic deletion, achieving the purpose of saving useful data to the greatest extent possible.

Embodiments of the disclosed technologies receive a first-party trained model and a first-party data set from a first-party system into a protected environment, receive a first third-party data set into the protected environment, and, in a data clean room, joining the first-party data set and the first third-party data set to create a joint data set for the particular segment, tuning a first-party trained model with the joint data set to create a third-party tuned model, sending model parameter data learned in the data clean room as a result of the tuning to an aggregator node, receiving a globally tuned version of the first-party trained model from the aggregator node, applying the globally tuned version of the first-party trained model to a second third-party data set to produce a scored third-party data set, and providing the scored third-party data set to a content distribution service of the first-party system.

The application discloses a data cleaning method and a terminal of a database, wherein slow query logs in a first database are monitored, and when the slow query logs exceed a preset number to generate an abnormal database table list, it indicates that there is a possibility of abnormal data in the database. A test database table is created in a second database according to the abnormal database table list, and test data corresponding to the data amount of the abnormal database table is added to the test database table. In this way, the deviation value of the size of the abnormal list can be determined according to the size of the abnormal list and the test list. When the deviation value reaches a threshold value, it is considered that there is a data hole in the list, and data cleaning is automatically performed for the abnormal list, thereby reducing the abnormal data stored in the database table and improving the overall system performance.

The embodiment of the application belongs to the field of artificial intelligence, and relates to a title generation method based on artificial intelligence, comprising the following steps: acquiring a pre-acquired title data set; cleaning titles in the title data set based on multiple data cleaning rules to obtain target text data; using the target text data as training data to train a pre-trained language model based on an adversarial training strategy and a noise reduction strategy, to obtain a title generation model; acquiring a target text to be recognized; inputting the target text into the title generation model, and processing the target text by the title generation model to generate a corresponding target title. The application also provides a title generation device based on artificial intelligence, a computer device and a storage medium. In addition, the application also relates to blockchain technology, and the target title can be stored in the blockchain. The application uses the title generation model to quickly and accurately generate the target title corresponding to the target text to be recognized, and ensures the accuracy of the generated target title.

The invention provides a database archiving method, system, equipment and program product applied to an online tourism transaction platform, which are applied to the technical field of computer data processing, and the method comprises the following steps: generating a data copy from to-be-archived business data in an online production database, executing archiving calculation in an offline analysis data storage area to determine a record identifier, and storing the record identifier in an offline analysis data storage area; the record identifier is sent to the asynchronous message queue, the record identifier is obtained through the data cleaning service, and the data cleaning operation is executed in the online production database. According to the scheme, data cleaning of the online production database can be achieved on the premise that the performance of an online transaction system is not affected, and the data cleaning efficiency is improved. And furthermore, the archiving integrity of the order associated data is ensured through audit-driven cascade calculation, so that data cleaning of an online production database can be realized on the premise of not influencing the performance of an online transaction system.

The application relates to a data cleaning method and device, computer equipment, a storage medium and a computer program product, and relates to the technical field of cloud computing. The method comprises the following steps: determining a cleaning cache space corresponding to each to-be-cleaned data table according to a data type corresponding to each to-be-cleaned data table, and storing index information of each to-be-cleaned data table in the corresponding cleaning cache space; determining a target cleaning cache space from a plurality of cleaning cache spaces, deleting data in a to-be-cleaned data table corresponding to the index information in the database based on the index information in the target cleaning cache space, and determining the target cleaning cache space from the remaining cleaning cache spaces, jumping to the step of deleting data in a to-be-cleaned data table corresponding to the index information in the database based on the index information in the target cleaning cache space until there is no target cleaning cache space. The method can improve the data cleaning efficiency.

The invention relates to a process knowledge graph error detection method and system, and the method comprises the steps: carrying out the data cleaning of a process knowledge graph, and constructing a training data set; initial embedding is generated for the entities and the relations of the triads based on a pre-training language model, attribute embedding of the entities related to the relations is aggregated based on an attention mechanism, and attribute-enhanced entity embedding is obtained; based on different multi-dimensional relation views, analyzing the triad embedded in the entity, and respectively calculating a credible score of each multi-dimensional relation view; calculating a comprehensive credibility score of the triad embedded by the entity based on each credibility score; and training an interval-based sorting loss function model by taking the triad and the comprehensive credibility score thereof as training data to form an error detection model, and analyzing the comprehensive credibility score by adopting the error detection model to judge the correctness of the triad. According to the invention, the accuracy of triple error recognition is improved.

Methods and systems for restricting queries to a database according to a privacy budget. The technology includes assigning a first privacy allowance to first data in a database table, the first privacy allowance being an amount of a privacy currency, and assigning a second privacy allowance to second data in the database table; receiving a query from a database user, the query including a specified amount of the privacy currency; and allowing processing of the query only when, for each one of the first data and second data that must be accessed to service the query, the specified amount of privacy currency is equal to or less than a remaining privacy allowance for the data.