509 results about "Data partitioning" patented technology

The invention relates to an experimental data processing method and device, an AI analysis module and computer equipment, and belongs to the field of data processing.The method comprises the steps that multi-dimensional original data are partitioned according to types, and formats are unified; generating a similarity matrix based on time and space neighborhood information, and marking abnormal fluctuation points; effective signals are separated through a time-frequency feature matching noise library; extracting multi-layer features of basic statistics, time sequence correlation and trend change; and dynamically screening core features to update the tracking type experimental model. The matched AI analysis module integrates hardware circuits of data partitioning, similarity calculation, anomaly marking, noise matching, feature extraction and model updating, and whole-process acceleration is achieved. According to the method, through multi-dimensional data compatibility processing, accurate anomaly detection, multilayer feature fusion and model adaptive optimization, the experimental data processing efficiency and conclusion reliability are remarkably improved, and the method is suitable for real-time analysis of multiple scenes such as scientific research and industry.

A method for facilitating supervised generative optimization for synthetic data generation is disclosed. The method includes receiving, via an application programming interface, inputs that include input data and parameters; partitioning the input data to generate data sets, the data sets including training data sets, validation data sets, and test data sets; tuning hyperparameters of synthesizers by using the data sets and supervised optimization that is based on downstream performance metrics; determining a mixture distribution from among the tuned synthesizers; training machine learning models based on the mixture distribution; and generating, by using the trained machine learning models, sets of synthetic data based on the input data.

The invention discloses a flame surface database construction method and device, equipment and a storage medium, and relates to the technical field of database management. Comprising the steps of performing data partitioning processing on a discrete post-mixed score space and a scalar dissipation rate in a target thermodynamic system based on a preset data partitioning method so as to obtain a data base point formed by a partitioned post-mixed score and a partitioned post-scalar dissipation rate; determining a constraint value set corresponding to the data base point according to a constraint flame surface equation corresponding to each constraint in the target thermodynamic system; solving the constraint potential energy form equation of each constraint in the constraint value set to obtain a corresponding constraint potential energy value, and determining the mass fraction distribution of each component corresponding to the data base point in the target thermodynamic system based on the constraint potential energy value, so as to construct a target flame surface database based on the temperature distribution and the mass fraction distribution corresponding to the data base point. In this way, the dimensionality of solving the equation set is reduced by reducing the number of the components of the control equation for describing the system to the number of the constraints.

The invention relates to a data platform file migration method, a computer program product and a data platform, and the data platform file migration method comprises the steps: scanning a to-be-migrated source end data file, and obtaining the type of the source end data file; if the type of the source end data file is in the state set of the target reinforcement learning model, taking the type of the source end data file as the current state of the target reinforcement learning model, and obtaining the current action of the target reinforcement learning model in response to the current state; using the current action of the target reinforcement learning model to block the source end data file; constructing a Merkel tree of the source end data file based on the blocking result of the source end data file; comparing Merkel trees of the source end data file with Merkel trees of the target end data file through layer-by-layer hash to locate difference data blocks; and migrating the difference data block to the target end. The problem that in an existing data platform file migration method, a data partitioning strategy is not reasonable, and consequently difference positioning is prone to failure is solved.

The invention provides an unstructured data synchronization method and system based on Flink, and relates to the technical field of big data process.The method comprises the steps that fragmentation processing is conducted on unstructured data, hierarchical storage of the data in a heap memory layer and a RocksDB layer is achieved on the basis of access feature scores, data partition optimization is conducted in combination with deep reinforcement learning, and the unstructured data are synchronized. And the parallelism degree parameter and the resource quota are dynamically adjusted by calculating the back pressure index value, and the data synchronization reliability is ensured by adopting a check point snapshot mechanism, so that the data processing efficiency and the system throughput are effectively improved.

The invention discloses an autoregressive language model training process-oriented data leakage risk quantification method, which comprises the following steps of: executing enhanced member data division processing on an original training text set, and generating a ternary partition text set containing member, non-member and key boundary texts through an optimization function; for each text, using the target model to extract member attributes from three channels of forward reasoning, back propagation and state evolution, and fusing the member attributes into member attribute vectors; inputting the member attribute vector into a hierarchical comparison embedded network, and mapping the member attribute vector into an optimized embedded representation vector through comparison learning including similar clustering, heterogeneous separation and boundary positioning; and inputting the embedded representation vector into a dynamic reasoning classifier capable of perceiving a training stage, judging the risk membership degree of the dynamic reasoning classifier, and generating a data leakage risk quantification result. According to the invention, real-time and fine-grained dynamic quantification can be carried out on the leakage risk, and timely early warning is provided for model training.

A system for a multi-tenant API framework in scalable cloud-based database solutions, which includes the following: an API layer module that enables access to database services for multiple tenants; a tenant management module configured to handle tenant authentication, authorization, and role-based access control; a data partitioning module for partitioning tenant-specific data into isolated database structures; a resource allocation module for the dynamic allocation of computing resources to individual tenants based on usage patterns; a security module configured to protect tenant data through encryption and secure data transmission; A monitoring and analysis module for real-time monitoring of tenant activities and system performance.

The invention provides a large-scale encrypted traffic frame-by-frame clustering analysis method based on a big data architecture, and relates to the technical field of big data, and the method comprises the steps: carrying out the data partitioning and storage of target encrypted data obtained through the preprocessing of original encrypted traffic data; based on a clustering visualization result obtained by visualizing a target clustering result obtained by carrying out frame clustering analysis on the target encrypted data, judging whether a data traffic abnormal behavior exists or not, and when the data traffic abnormal behavior exists, generating an abnormal analysis report; and an abnormal analysis report is transmitted to safety management personnel so as to take corresponding measures in time for defense. The method comprises the following steps: visualizing a target clustering result obtained by carrying out frame clustering analysis after processing and partition storage on encrypted traffic data based on a big data architecture, identifying traffic data exception, generating an exception analysis report when the exception exists, and transmitting the exception analysis report to safety management personnel to take corresponding measures for defense. The network threat identification capability is effectively enhanced, and the overall protection level of network security is further improved.

The invention discloses a data quality management method and system, belongs to the technical field of data management, and aims to solve the technical problems of how to improve the data quality management effect and improve the data service quality. Comprising the following steps: during data acquisition, performing quality detection through a corresponding quality detection rule, and performing data partitioning and data labeling according to an initial data quality detection evaluation result; performing corresponding preprocessing on the data of the to-be-used data partition according to the initial data quality detection and evaluation result; evaluating data quality according to the data preprocessing result to obtain an evaluation result, and adjusting and optimizing a quality detection rule according to the evaluation result; a user is supported to evaluate data quality through a data quality evaluation service, a data problem report rate of the user is formed, the data quality is evaluated according to the data problem report rate of the user, and data acquisition quality detection and data integration preprocessing operation are improved.

The invention relates to a firmware upgrading method and electronic equipment. The method comprises the following steps: in a first stage of upgrading, constructing upgrading meta-information according to a current partition table and a to-be-upgraded firmware data storage area base address obtained by calculating to-be-upgraded firmware meta-information, and according to a block list description file, updating the to-be-upgraded firmware data storage area base address; in the first-stage upgrade, a partition table file with a preset name in a firmware upgrade package under a preset firmware upgrade package path of an encrypted preset user data partition and all to-be-upgraded firmware mirror image files are stored in an address space determined by a base address of a to-be-upgraded firmware data storage area; and determining a to-be-upgraded firmware data storage space end address and a to-be-upgraded firmware data moving base address according to the guide communication partition address space change mark, and copying data in an address space determined by the to-be-upgraded firmware data storage space end address to the to-be-upgraded firmware data moving base address. According to the invention, under the condition of user data partition encryption, the firmware with any change of the partition table can be safely and reliably upgraded.

The invention relates to the technical field of data synchronization, in particular to a data synchronization method and system for communication software, which comprises the following steps of: counting the network bandwidth utilization rate of the communication software, the synchronization request frequency of a user and the synchronization request size, calculating the transmission demand quantity of each data partition, comparing the current load capacity of a different partition with the network transmission capacity, and determining the network transmission capacity of the different partition. And adjusting the transmission range and the partition boundary of the data partition, and generating a dynamic partition transmission range value. According to the method, the network bandwidth utilization rate, the user request frequency and the request size are comprehensively calculated, the partition transmission range and boundary are dynamically adjusted, network resource allocation is optimized, the influence of high-load partitions on the overall transmission capacity is reduced, data items in the partitions are classified and processed based on the updating frequency and the access mode, and conflict frequency statistics and sorting are combined, so that the transmission capacity of the high-load partitions is improved. High-conflict data items are solved preferentially, the synchronization delay amount is evaluated and subjected to threshold judgment, key data are transmitted preferentially, and the timeliness and consistency of high-sensitivity data are improved.

The invention relates to the technical field of big data processing, and particularly discloses a computer data management system based on big data. The system comprises a distributed data acquisition module which captures multi-source data through an API gateway and a crawler and embeds a quality label; the intelligent storage scheduling engine is used for realizing automatic migration of cold and hot data by adopting a column-type and distributed hybrid architecture; fusing a computing framework, and dynamically coordinating flow processing and batch processing resource execution feature extraction; the self-adaptive strategy center is used for generating data partitions and encryption strategies based on reinforcement learning; and a multi-level security protection system is adopted, and attribute base decryption and dynamic desensitization are deployed. The problems of resource scheduling rigidity and security protection lag in the prior art are solved, the storage cost is reduced by more than 40%, the query delay is compressed to be within 200ms, and the method is suitable for real-time decision-making scenes in the fields of e-commerce and finance.

The invention discloses a cable defect analysis method and system in a cable laying process, and the method comprises the steps: dividing a current data sequence according to a preset data division strategy, obtaining at least one current data sub-sequence, and obtaining cable images of different monitoring positions according to the at least one current data sub-sequence; preprocessing the cable image to obtain a target cable image, and performing cable contour and background region segmentation on the target cable image to obtain a cable contour image; calculating a cable area based on the cable contour image, and extracting cable surface features; analyzing the surface features of the cable based on a set noise threshold, and removing noise features to obtain optimized surface features of the cable; and analyzing the surface features of the optimized cable based on the defect detection model, and outputting surface defect information. The accuracy and the real-time performance of defect detection are remarkably improved, the manual inspection cost and the leakage detection risk are reduced, and reliable guarantee is provided for safe operation of the cable.

The invention discloses a database data storage system based on multi-level cache, and relates to the technical field of database data storage systems, the database data storage system comprises a basic storage layer, a first-level cache layer, a second-level cache layer, a cache cooperative control module and a data read-write scheduling module, the basic storage layer is used for persistently storing total data of a database; the first-level cache layer is divided into a plurality of hot data partitions; and the second-level cache layer is provided with quasi hot data partitions in one-to-one correspondence with the hot data partitions. According to the database data storage system based on the multi-level cache, the system constructs a'high frequency-secondary high frequency-full quantity 'layered architecture through the first-level cache, the second-level cache and the basic storage layer, dynamic flow of data is achieved in cooperation with a data aging counter and a heat recovery detector, heat reduction data is migrated from the first-level cache to the second-level cache, and the heat reduction data is transferred to the second-level cache. And the heat recovery data returns to the first-level cache from the second-level cache to ensure that the high-frequency data is always kept in a cache layer.

The invention provides a snapshot generation method and device, electronic equipment and a storage medium, and the snapshot generation method comprises the steps: responding to any data partition update version, sending a version identification obtaining request to a version module, and taking the version identification returned by the version module as the version identification of the updated version of the data partition, the version module is used for generating a version identifier in a unique and incremental manner when receiving a version identifier acquisition request every time; and receiving a snapshot generation instruction, obtaining a version identifier newly generated by the version module, and generating a snapshot based on the snapshot version of each data partition, the snapshot version being a version to which the version identifier having the smallest difference with the newly generated version identifier belongs.

The invention discloses a database self-adaptive mixed data partitioning method and system based on reinforcement learning, and belongs to the field of artificial intelligence. According to the method and the device, the problems that a large amount of redundant data needs to be accessed and self-adaptive data partitioning cannot be automatically performed according to the dynamic working load when the query load is executed in an existing data partitioning scheme are solved. According to the method, a workload prediction model is established, the occurrence frequency of various types of queries in query workloads of a database in a future period of time is predicted based on historical workloads, a mixed data partition problem is modeled as a Markov decision process, and the state, action and reward of an intelligent agent are defined; according to the predicted occurrence frequency of various types of queries and the trained reinforcement learning model, an appropriate mixed data partitioning scheme is adaptively recommended for the arriving query workload, and the amount of data needing to be accessed when the query load is executed is reduced. The method can be applied to self-adaptive mixed data partitioning of the database.

The invention discloses a USB flash disk encryption system for data element circulation. The USB flash disk encryption system comprises a client and a USB flash disk, the client comprises a USB flash disk content display interface and a key input module; the USB flash disk comprises a connecting interface, a data encryption module and a storage chip; the data encryption module comprises a data partitioning unit, a key generation unit and an AES encryption unit; the data partitioning unit is used for partitioning original data into data blocks with odd numbers and even numbers; the key generation unit is used for expanding the original key into a plurality of round keys; the AES encryption unit comprises a first AES encryption core and a second AES encryption core; the first AES encryption core uses a plurality of round keys to perform multi-round encryption processing on each data block with the odd number, and the second AES encryption core uses a plurality of round keys to perform multi-round encryption processing on each data block with the even number in sequence. According to the invention, the data encryption efficiency of the USB flash disk can be improved, and the data encryption security is enhanced.

The invention provides a data query method which can be applied to the technical field of big data. The data query method comprises the following steps: in response to received user input information, identifying the type of the input information; under the condition that the type of the input information is the first type, determining a keyword according to the input information; performing association matching on nodes in the knowledge graph based on the keywords to obtain entity information; wherein the knowledge graph is used for reflecting structured representation of data lake content; querying target information in a pre-constructed data lake according to the entity information; wherein the data lake comprises a plurality of data partitions, the sources of the data in each data partition are different, the data in different data partitions have an association relationship, and the target information comprises the data with the association relationship in the plurality of different data partitions. The invention further provides a data query device and equipment, a storage medium and a program product.

The invention discloses a data scheduling method, device and system and a storage medium. The method comprises the steps of obtaining scheduling duration required by to-be-scheduled data in a service system in a data link of a data warehouse; dividing the plurality of time zones into a plurality of time zone segments according to the scheduling duration; determining scheduling start time in a preset reference time zone according to the minimum time zone of each time zone segment; determining a target time period for dividing the to-be-scheduled data in a preset reference time zone according to the scheduling start time; for each time zone segment, determining the data of which the generation time is within the target time period in the to-be-scheduled data of each time zone segment as target data; and scheduling the corresponding target data to the data warehouse according to the scheduling start time of the time interval segmentation, so as to store the target data to the data partition, corresponding to the target time interval, of the data warehouse. According to the scheme, under the condition that the time-zone-crossing data scheduling timeliness is ensured, the performance overhead during scheduling is reduced, and the data consistency is ensured.

The invention relates to the technical field of industrial big data processing and algorithm scheduling, and particularly discloses an industrial algorithm model scheduling method and system for industrial big data, and the method comprises the steps: constructing an algorithm dependency graph of an execution pipeline composed of a plurality of industrial algorithms, bottleneck nodes influencing the overall performance are identified through a critical path analysis algorithm; carrying out operation fusion on algorithm pairs with close dependency relationships; optimal data partitioning strategies are determined for different algorithms, and an elastic execution pipeline is realized; allocating the most matched heterogeneous computing resources for the algorithm operation on the critical path, and adjusting the execution priority of the algorithm operation in real time according to the system load and the operation scene; a fine-grained synchronization system is realized; the industrial algorithm execution efficiency is improved, and the high-performance requirement of an industrial big data processing scene is met.

The invention provides an efficient query acceleration and cache optimization device for data knitting, which comprises a semantic partitioning component used for performing dynamic partitioning on a multi-source heterogeneous data set based on multi-source heterogeneous semantic metadata and embedding cross-domain association identifiers of data knitting to generate a semantic partitioning index table; the query intention mapping component is used for executing intention extraction on a query request initiated by a user to generate a query intention representation, and establishing a mapping relationship of the query intention representation to-be-queried target data partitions according to the semantic partition index table so as to generate a query-partition mapping table; the cache dynamic sorting component is used for generating a cache priority sorting table according to the query-partition mapping table and historical query frequency statistical data; and the scheduling and query execution component is used for optimizing the cache mechanism of the to-be-queried target data partition based on the cache priority sorting table and a preset proportion threshold value. According to the method, the time consumption of cache calling and source data repeated loading in the later query period is reduced.

A method and a system for utilizing dynamic data partitioning to ensure consistency of event publications with associated business transactions in a distributed microservices architecture in order to optimize efficiency in polling data are provided. The method includes: defining a hash space that includes a range of assignable hash values; deploying a respective instance of each microservice to form a cluster of microservices within the distributed microservices architecture; allocating a respective subset of the hash space to each microservice; and facilitating a data polling capability with respect to a data table based on the allocated respective subset of the hash space and the deployed respective instance for each microservice.

The invention discloses a data processing method, device and equipment and a computer readable storage medium. The method comprises the following steps: obtaining transaction data in a target period, the transaction data comprising first transaction data updated in the target period; determining a target first-level partition corresponding to the first transaction data and a target hot data partition under the target first-level partition from a plurality of first-level partitions of the data warehouse based on the transaction occurrence period of the first transaction data; the first-level partition is determined based on a transaction generation period, a second-level partition is arranged under the first-level partition, the second-level partition comprises a cold data partition and a hot data partition, the cold data partition is used for storing newly added transaction data in the transaction generation period, and the hot data partition is used for storing updated transaction data in the transaction generation period; and updating the second transaction data in the target hot data partition under the target first-level partition by using the first transaction data. According to the data processing method provided by the embodiment of the invention, the data updating efficiency can be improved.

The invention discloses an aerospace equipment performance evaluation method based on digital test data fusion, and belongs to the field of electronic engineering and computer science, and the method comprises the steps: designing a physical test and a digital test, selecting related indexes of aerospace equipment, and carrying out corresponding physical and digital tests around the related indexes of the aerospace equipment; performing preprocessing and multi-source data integration on the physical test data; performing data quality evaluation and virtual data division on the digital test data; and performing double fusion on the result through an adaptive fusion algorithm to obtain a performance evaluation result. The advantages of the digital test and the physical test can be fully exerted, and comprehensive and accurate evaluation of the aerospace equipment performance is realized through deep fusion of the digital test data and the physical test data.

The invention discloses a data storage system and method for artificial intelligence deep learning, and relates to the technical field of data storage, and the system comprises a data obtaining module which is used for recognizing a data source and obtaining data from the data source; the data preprocessing module is used for automatically detecting and cleaning abnormal values in the data; the data storage design module is used for automatically storing the data in different types of storage media based on the access frequency and importance of the data and establishing a dynamic data partition model; the real-time data processing module is used for storing and managing real-time data streams; and the data backup module is used for backing up the data. According to the method, intelligent storage layering is provided, the data can be automatically stored in different types of storage media according to the access frequency and importance of the data, the storage cost is optimized, the access speed is increased, and a data partitioning strategy can be adjusted in real time to cope with the change of the data volume by establishing a dynamic adjustment model and adopting a dynamic data partitioning technology.

The invention discloses a dynamic partition-based tax form intelligent asynchronous processing and real-time query method, which comprises the following steps of: in response to multiple groups of tax form data, analyzing a characteristic field numerical value to generate a unique request identification code; carrying out numerical processing on the feature field to generate a dynamic partition sub-identifier, and aggregating to generate a dynamic partition combination identifier; matching a target data partition corresponding to the dynamic partition combination identifier through a constructed dynamic partition mapping table, and importing structured form data to form a dynamic partition database; constructing an asynchronous task key value pair, writing the asynchronous task key value pair into an asynchronous message queue, regularly scanning the asynchronous message queue, and when a to-be-processed task is detected, identifying whether the task queue contains a priority mark or not, and dynamically adjusting the execution priority of the task queue; and implementing multi-dimensional tax data query from the dynamic partition database based on real-time query conditions and the optimized task queue. The tedious and low-efficiency manual operation is avoided, and the processing efficiency of the form and the tax data is improved.

The invention discloses a localization-adaptive track point visual rendering method based on partitioning and hierarchical caching, and the method comprises the steps: S1, inputting a track data stream, carrying out the reading according to the source type of the data stream, and carrying out the data partitioning and LOD processing of the track point data; s2, performing data analysis and rendering on the data; s3, carrying out batch processing on GPU vertexes, according to the state of the circular buffer, if the vertexes are writable, incrementally writing vertex data, and if Wrap is needed, carrying out loopback writing and updating a pointer; and S4, finally, the descriptors and the styles are updated, rendering increment drawing is carried out on the track points, and visual display is carried out. The method supports real-time rendering and playback of ten-million-level track points, has visual and instantiated rendering of window dynamic loading, space-time partitioning, hierarchical caching and WebWorker decoupling calculation, and improves the space-time data rendering capability of the domestic autonomous controllable field by ten thousand times; and the user experience is greatly improved by a non-perceptual interaction and transition smoothing algorithm.

The invention relates to the technical field of data cleaning, in particular to an automatic heterogeneous data cleaning method and system based on a configurable rule engine. The method comprises the following steps: acquiring a heterogeneous data source, uniformly converting a data mode and an instance into a multi-dimensional heterogeneous fact graph, calculating an execution weight for each conflict rule according to the authority of the data source and the priority of the rule when a write operation conflict is detected, segmenting data partitions by utilizing the betweenness centrality of a connected component and a node, the method comprises the following steps of: generating a plurality of data partitions, forming rule subsets corresponding to the data partitions, creating isolated Drools session instances to perform parallel cleaning, generating traceability logs in the parallel cleaning process, aggregating the traceability logs after all the data partitions are cleaned, and re-evaluating and updating the authority of each data source to solve rule conflicts in the next cleaning period. According to the scheme, the writing conflict can be precisely processed, the safety of parallel cleaning is improved, the processing period is shortened, and a continuously optimized processing closed loop is formed.

The invention discloses a heterogeneous perception traffic prediction method based on federated learning, and the method employs federated learning to design a unified heterogeneous perception framework, and supports an existing centralized traffic prediction model. Clients with similar traffic flow data distribution are gathered together by utilizing multi-dimensional positive sample comparative learning, so that the clients of the same kind can cooperatively train a model, and the influence of data isomerism between different clients is avoided; the model of each stage is trained in sequence based on a time window by using data partition, so that the influence of data missing on traffic prediction is reduced; noise detection is used for global detection and local denoising, so that the quality of client data is ensured.

The invention relates to a lossless database watermarking scheme which comprises the following steps: S1, inputting original data into a preprocessing module for preprocessing, and performing attribute selection on the original data to obtain preprocessed data; s2, processing the preprocessed data in a watermark embedding module to obtain embedded data; s3, if the embedded data is not maliciously attacked, directly entering mask generation and watermark extraction in a watermark extraction module for processing, and then performing majority voting processing to generate a watermark; and S4, if the embedded data is maliciously attacked, performing data partition processing in a watermark extraction module, performing mask generation and watermark extraction processing, and finally performing majority voting processing to generate a watermark. The overall system provided by the embodiment of the invention is zero in distortion, does not modify the original database while embedding the watermark information into the database, is suitable for application scenes needing high-precision data, and has good robustness.