117 results about "Data sanitization" patented technology

The invention relates to the technical field of intelligent decision making, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a decision making method, device, equipment and medium based on multi-modal data, comprising: collecting and cleaning multi-modal original data, executing format conversion and desensitization processing, and generating standardized data; multi-modal features are extracted and fused based on the standardized data, and multi-modal feature vectors are generated; performing reasoning on the multi-modal feature vector through a reasoning model to generate reasoning result data; pushing the reasoning result data to a review terminal, and receiving correction feedback data; and updating parameters of the reasoning model based on the corrected feedback data, and generating an updated reasoning model. According to the method, after data cleaning, format conversion and desensitization processing, multi-modal feature extraction and reasoning model reasoning processing are executed, effective fusion and accurate reasoning of data multi-modal features are achieved, and decision accuracy and data safety are improved.

The invention discloses an intelligent bid evaluation expert extraction and management system, and relates to the technical field of expert extraction systems. Comprising a data cleaning and compression module, a database logic and table relation design module, an expert information collection standard and automation module, an information real-time updating and principal management module, an extraction algorithm design module, a safety and authority management module and a real-time monitoring and auditing function module. Through a self-adaptive sparse auto-encoder and a dynamic clustering technology in the data cleaning and compression module, abnormal values in a bid evaluation expert database can be detected and cleaned, data compression is performed by adopting a self-adaptive dictionary coding algorithm, and the storage efficiency and query performance of the database are improved; based on a multi-modal information fusion and reinforcement learning method, accurate collection, dynamic updating and intelligent management of expert information are realized. The randomness and fairness of the bid evaluation expert extraction process are ensured by using a mode of combining deep reinforcement learning and a quantum random number generator in an extraction algorithm design module.

The invention discloses a multi-scene application data acquisition method based on atomization design. The method comprises the following steps: analyzing reported data into an atomization data structure; performing semantic recognition on the data labels to generate a label semantic mapping result; constructing a layered cache structure; generating a compressed semantic mapping rule by applying a multi-level semantic difference coding algorithm; identifying an optimal query path from the tag affinity matrix; and executing historical data cleaning based on the data value. Intelligent mapping of data labels is achieved through semantic recognition and version control, the storage efficiency is improved through predictive lazy loading and compression coding, the query path is optimized based on the affinity matrix, and the technical problem of multi-scene application data collection is effectively solved.

The invention provides a multi-dimensional accounting data integration and analysis system, and belongs to the technical field of accounting information processing, the multi-dimensional accounting data integration and analysis system comprises a data acquisition system, a data source interface unit of the data acquisition system can be connected with a plurality of data sources, and an acquisition control unit manages acquisition; the data integration system comprises a preprocessing module (format conversion, data cleaning and fusion modules); the data analysis system is provided with an analysis dimension setting module, an algorithm execution module and a result visualization module; the data storage system comprises a storage architecture management and index module, solves the problems that a traditional method is difficult in data source integration and single in analysis dimension, can effectively integrate multi-dimensional accounting data, improves the comprehensive analysis capability, improves the data security and query performance, and visually presents an analysis result to facilitate decision making.

Systems, computer program products, and methods are described herein for secure training of machine learning models. The present disclosure is configured to receive a training dataset, wherein the training dataset comprises personally identifiable information (PII); implement a data sanitization algorithm using adjustable data sanitization parameters on the training dataset to generate a corresponding sanitized training dataset; train an ML model using the sanitized training dataset; deploy the trained ML model on a live dataset; evaluate a performance accuracy of the ML model against an accuracy threshold; and iteratively adjust the data sanitization parameters for subsequent iterative re-training of the ML model until the performance accuracy of the ML model meets the accuracy threshold.

Systems and methods for intelligent data sanitization employing PCM and AI / ML are provided. The idea uses AI / ML to detect specific facts that needs sanitization rather than full properties in incoming records. Data sanitization is optimized using this focused method, saving computational resources. To properly manage changing data volumes, PCM shifts between Logical 0 and Logical 1 states. Logical 0 processes smaller volumes with high resistance and low conductivity, while Logical 1 processes large volumes with low resistance and high conductivity. The AI / ML module organizes and directs data to maximize resource and processing efficiency. The PCM processes data in-memory and directly overwrites, eliminating erasure. AI / ML and PCM integrate to sanitize data quickly, efficiently, and securely, improving system performance and data integrity without a central repository. The system dynamically adjusts to changing data patterns, protecting and optimizing data.

The invention provides a building structure intelligent design method and system based on 3D modeling, electronic equipment and a storage medium, and relates to the field of building intelligent design, and the method comprises the following steps: obtaining three-dimensional data through an unmanned aerial vehicle, oblique photography and laser scanning, and carrying out data cleaning, conversion and screening; performing feature extraction and semantic segmentation on the point cloud data by using a deep learning model; performing intelligent registration and fusion on the two types of point clouds by adopting a deep learning model; a three-dimensional model is generated based on the registration data, a parameter form is generated in combination with the building planning data, and a component model is constructed; collision detection, smooth verification and mechanical simulation are carried out on the whole model, parameter adjustment and optimization are carried out according to verification results, through automatic data collection, multi-source point cloud fusion, intelligent parameter modeling and multi-dimensional verification and optimization, the problems that a traditional method is low in efficiency and insufficient in precision are solved, and the design efficiency and precision are remarkably improved.

The invention provides a large-scale feature data storage and query method. The large-scale feature data storage and query method comprises the following steps: S1) feature data storage: writing real-time feature data into an original column cluster of HBase and retaining a preset number of historical versions; s2) regularly merging historical data: regularly merging historical version data in the original column cluster through a batch processing task, and writing the merged data into a new summary column cluster; s3) feature data query: when feature data is queried, preferentially reading data from the summarized column cluster, and if the data is not found, returning to the original column cluster for query; and S4) dynamic scheduling and data cleaning: dynamically adjusting a feature data merging strategy according to the access frequency of the feature data, and regularly cleaning out-of-date data.

The invention relates to the technical field of network communication, and discloses a cache processing method and device of a gateway router and network device.The method comprises the steps that front-end equipment temporarily stores data in local storage after a user submits the data, the front-end equipment and the gateway router continuously temporarily store the data in the network interruption period, and when network recovery is detected, the front-end equipment and the gateway router continuously store the data in the local storage period; the data uploading logic is automatically triggered, the front-end equipment sends the temporarily stored data to the gateway router, and after the data is successfully uploaded to the server, the front-end equipment and the gateway router automatically clean the temporarily stored data; the device comprises front-end equipment, a gateway router, a server and a data cleaning module, the network equipment comprises a processor and a memory which is in communication connection with the processor, wherein a computer program which can be executed by the processor is stored in the memory. By adopting the technical scheme of dynamic bandwidth regulation and control and data transmission rate adjustment, the technical effect of optimizing the data transmission rate in real time according to the network state is achieved.

The invention relates to the technical field of system database management, in particular to an intelligent monitoring analysis method and system for gate opening and closing working condition data, and the method comprises the steps: obtaining fault records from a historical fault data storage library of a specified gate, and constructing a fault feature library; acquiring historical monitoring data of the sensor group, performing pattern recognition and statistical analysis to form a fault feature weight table, and storing the fault feature weight table in a data cleaning database; and classifying the sensor group through a database index technology by utilizing the fault feature weight table, the external prediction data and the sensor installation position data, and storing a classification result into a sensor level index table. And a distributed data intelligent monitoring system is formed through N independent data processing units in combination with a database interface protocol. The data processing unit comprises a data acquisition layer, a data real-time analysis layer, a data cleaning and quality control layer and a data storage and retrieval layer, and high-efficiency processing and retrieval of data are ensured.

The invention discloses a financial reconciliation system reconciliation risk identification control method and system, a medium and equipment. The method comprises the following steps: a payer acquisition step: acquiring a transaction bill of a payment channel; a transaction party acquisition step: acquiring a transaction bill of a business party; an account checking platform acquisition step: acquiring an account checking result generated by the financial account checking platform; a risk identification control step: checking the transaction bill of the payment channel, the transaction bill of the business party and an account checking result generated by the account checking platform, modifying error data, and sending a processing result to an abnormal data processing module; an abnormal data processing step: receiving an error data modification structure, and manually modifying the unmodified data; and an abnormal data display step: analyzing, counting and displaying the reconciliation data subjected to error data processing. According to the method, the collection and automatic reconciliation of the transaction bill can be realized, and the abnormal data generated after reconciliation can be cleaned and corrected.

Embodiments of the present disclosure may provide a data clean room allowing secure data analysis across multiple accounts, without the use of third parties. Each account may be associated with a different company or party. The data clean room may provide security functions to safeguard sensitive information. For example, the data clean room may restrict access to data in other accounts. The data clean room may also restrict which data may be used in the analysis and may restrict the output. The overlap data may be anonymized to prevent sensitive information from being revealed.

The invention relates to the technical field of thermal power plant abnormal data processing, in particular to a thermal power plant production monitoring system network security data set abnormal data identification and cleaning method, which comprises the following steps: mapping a spatial-temporal feature vector to a preset thermal power plant multi-level causal graph to obtain a data causal graph; performing information propagation and node updating on the data causal graph, and calculating an abnormal score of each node in a preset thermal power plant multilevel causal graph; identifying abnormal data according to the abnormal score and the network security data set; and cleaning the abnormal data according to the abnormal type and a preset data cleaning strategy. According to the method, correlation modeling of data on time and a system topological structure is realized through a multi-level cause and effect graph; an abnormal score is calculated through information spreading and node updating, and an abnormal event in the network security data can be accurately identified; the abnormal data is cleaned based on the preset data cleaning strategy, redundant, wrong and abnormal data can be effectively removed, and meanwhile key risk information is reserved.

Methods and systems are described herein for leveraging artificial intelligence to sanitize sensitive data and prevent the data from leaving the mobile device and / or be exposed to unauthorized third parties. More specifically, methods and systems are described for a novel and unconventional architecture for a data sanitization application, a novel and unconventional delivery format for the data sanitization model, and a novel and unconventional output format of the data sanitization model.

A system and method for automated bulk creation of Universal Resource Locators (URLs) with embedded Urchin Tracking Module (UTM) parameters is disclosed. The method processes input data, including endpoint URLs and UTM parameters, through pre-processing steps such as data cleansing and format standardization. The method appends UTM parameter-value pairs to endpoint URLs, optionally encoding the parameters. The generated tracking URLs are stored in a structured database for seamless integration with analytics tools, enabling reliable tracking and actionable insights into web traffic and user behavior.

The invention relates to the technical field of deep learning, in particular to a large model knowledge base question answering method and system with a timeliness map. The method comprises the following specific steps: constructing a graph database capable of updating data in time and regularly performing timeliness data cleaning; inputting contents in a natural language, and performing triple extraction through a natural language large model; obtaining a specific query result from a graph database and a vector library according to the obtained triple result; the natural language large model carries out reference answering according to the graph data and the query result of the vector library; the method has the advantages that the data in the knowledge graph has the capacity of updating along with time change, the expired data in the knowledge graph can be deleted in time, and the timeliness value of the graph data is improved.

The invention discloses a packet management method and device, equipment and a storage medium, and the method comprises the steps: obtaining a processing instruction for a target packet, and publishing the target packet in a version control system Git; executing the processing instruction to perform at least one of installation, data cleaning and removal on the target packet from the Git; and monitoring the processed target packet to obtain a monitoring result and outputting the monitoring result. Therefore, the packet management scheme provided by the embodiment of the invention can be deeply integrated with the Git, so that packet management can be realized by virtue of a private packet management function supported by the Git without depending on a public network packet manager, thereby reducing the risk of information leakage and improving the information security. Moreover, packet management is realized by means of Git, and additional privatized packet management does not need to be established or additional server hosting does not need to be configured, so that the cost can be reduced.

A system for the near real-time exchange of regulatory-compliant and privacy-compliant consumer credit data between two parties (i.e., a credit bureau and a financial services institution) that facilitates the financial services institutions (i.e., lenders) to directly identify audiences of pre-screened consumers (i.e., prospects) to offer credit products without requiring the intervention of a third party, such as an agent of the bureau, employs a data clean room to facilitate the necessary data exchange. The clean room enables and expedites the presentment of firm offer of credit pre-screened offers across authenticated or authenticatable offline and digital online channels, thereby making credit available to consumers in a more timely and efficient manner while protecting consumer privacy.

The invention relates to the technical field of logistics personnel operation tracking, and discloses an operation timeliness monitoring and tracking method for logistics personnel. Comprising the steps of setting express time efficiency assessment information, and obtaining the express time efficiency assessment information; obtaining express service process information, extracting data resource information and data resource list information in the express service process information, and carrying out data processing on the data resource information and the data resource list information to generate express service aging data volume information; obtaining express service time-efficient data volume information and data storage principle information after data cleaning, and generating a database table structure according to the express service time-efficient data volume information and the data storage principle information; establishing a data analysis platform through a database table structure; the system has the advantages of facilitating real-time tracking of cargo positions and mastering of cargo operation conditions.

The invention particularly relates to a modeless data storage system supporting asynchronous change based on a hybrid storage structure. The modeless data storage system supporting the asynchronous change based on the hybrid storage structure comprises a storage module, a metadata management module, a data recombination module, a data cleaning module and a workload monitoring module. According to the mode-free data storage system supporting asynchronous change based on the mixed storage structure, services are provided for the outside through the mode storage system, asynchronous mode change is automatically triggered when write-in data is not matched with a current data mode, high-performance access of mass data can be met, flexible and free automatic data mode change can be achieved, and the data storage efficiency is improved. And the performance and the stability of the system are ensured.

The invention discloses a data processing method and device, equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: judging whether a current service table meets a preset table rotation condition or not based on the service table information of the current service table; under the condition that the current service table meets a preset table rotation condition, judging whether data cleaning of the data in the empty table is completed or not; and under the condition that data cleaning of the data in the vacant table is completed, taking the vacant table as a new current service table to respond to the service request. According to the invention, under the condition that the current service table meets the preset table rotation condition and the data in the vacant table is cleaned, the vacant table is used as the new current service table to respond to the service request. Compared with an existing mode of cleaning the data table in the service low-peak period, the method has the advantage that the table data can be cleaned under the condition that the online service is not paused by taking the cleaned empty table as the new current service table to respond to the service request.

The invention provides an expressway business form extraction method and system based on a neural network, and the method comprises the steps: obtaining to-be-processed sub-business forms, carrying out the data cleaning, and generating rule data; performing compliance verification on the rule data based on the agreed rule, and if the verification is qualified, storing the rule data as a standard business form according to the category; traversing each standard service form in sequence, identifying coincident entries, taking the coincident entries as association keys, and extracting a high-speed service form set belonging to the same user based on the association keys; and inputting the high-speed business form set into a pre-trained statement extraction model to generate a passing behavior statement. According to the invention, by integrating the expressway business forms, standardized processing and efficient management of data are realized. According to the method, the data compliance is ensured, the business forms of the same user are identified and associated, the traffic behavior statements are generated by using the statement extraction model, and convenient and accurate information support is provided for expressway business management.

The invention provides a prevention and control method and device for network-related illegal behaviors and a storage medium. The prevention and control method comprises the steps of obtaining network-related risk data; performing data cleaning, data conversion and feature selection on the network related risk data to obtain preprocessed multi-source heterogeneous data; inputting the multi-source heterogeneous data into a network-related susceptible population classification model, and determining a network-related susceptible population big data portrait; wherein the network-related susceptible population classification model is constructed based on a random forest algorithm; determining the network-related susceptible crowd according to the network-related susceptible crowd big data portrait; and the risk information of the network-related susceptible crowd is provided for related institutions. According to the method, deep acquisition, fine analysis and scientific modeling are carried out on the multi-source heterogeneous data, accurate identification and early warning of novel network-related susceptible crowds are realized, effective clues and laws hidden behind complex data are effectively identified, and the working efficiency of illegal risk disposal is improved.

The application discloses a systematic color SVG icon dataset construction method and device, and the method comprises the following steps: collecting SVG files, and performing data cleaning on the collected SVG files to obtain initial files; wherein the SVG files comprise SVG icons of different categories, and the initial files are color SVG files; performing standardization processing on the boundary box size of the initial files to obtain sample files; classifying the sample files, and dividing training set, verification set and test set according to the classified sample files to obtain SVG datasets. The application can improve the construction quality of color SVG datasets.

Systems and methods for intelligent data sanitization employing PCM and AI / ML are provided. The idea uses AI / ML to detect specific facts that needs sanitization rather than full properties in incoming records. Data sanitization is optimized using this focused method, saving computational resources. To properly manage changing data volumes, PCM shifts between Logical 0 and Logical 1 states. Logical 0 processes smaller volumes with high resistance and low conductivity, while Logical 1 processes large volumes with low resistance and high conductivity. The AI / ML module organizes and directs data to maximize resource and processing efficiency. The PCM processes data in-memory and directly overwrites, eliminating erasure. AI / ML and PCM integrate to sanitize data quickly, efficiently, and securely, improving system performance and data integrity without a central repository. The system dynamically adjusts to changing data patterns, protecting and optimizing data.

There is provided a data preparation function in a wireless communication network, the data preparation function comprising: one or more processors arranged to: collect data from one or more data sources in the wireless communication network; analyse the collected data to derive one or more data characteristics and to identify whether the collected data face one or more quality issues or irregularities; and prepare the collected data based on the analysis, including performing one or more of the following: data recovery to recover data missing from the collected data; data cleaning of the collected data; formatting of the collected data; or separation of the collected data into different data sets for one or more training tasks.

The embodiment of the present application provides a kind of data cleaning method and device, the method includes determining data attribute field from each business field of business scene, by data attribute field, determine the search field set from each database table, and determine the database table set matched with search field set, the source data associated with business scene is analyzed, determine the business data value of the business field to be cleaned, according to search field set and database table set, the structured query language sql data cleaning script corresponding to the business data value of the business field to be cleaned is constructed, and the relevant business data in the database table set associated with the business data value of the business field to be cleaned is cleaned by sql data cleaning script.It is thus, the scheme can realize the accurate cleaning of dirty data automatically by the constructed sql data cleaning script, so as to effectively reduce the time consumed by test personnel for cleaning dirty data.

The present invention relates to the field of data processing, and in particular to a method, device, computer equipment and storage medium for real-time synchronization of multiple tables in a data lake. The method comprises: when receiving an upstream table information change request, blocking the input and output data streams of the data lake; performing data cleaning on the cached data streams, and updating the table information that needs to be changed in the storage space according to the table information change request; after the table information update is completed, restoring the blocked input and output data streams. This allows the data lake to be hot-updated online without restarting the task to support the frequent table information update requirements of the upstream.

The present invention relates to a computer implemented method for data cleansing. In the method, a data pre-processing module of a first computing instance (3) obtains an at least two-dimensional un-shuffled data structure (10). The data structure (10) comprises healthcare patient data. The data structure (10) comprises data on a plurality of patient records. Each patient record comprises one or more fields wherein each field has a field type and wherein the patient records have a pre-shuffled order. The data pre-processing module of the first computing device shuffles fields of a first field type by moving fields having the first field types from one patient record to another patient record. Fields with the first field type have a post-shuffled order after the shuffling thereby creating a shuffled data structure(lO). A communications module transmits (5) the shuffled data structure (10) comprising the healthcare patient data over a communications network (7) from the first computing instance to a second computing instance.