2470 results about "Data migration" patented technology

The invention relates to an intelligent computing power and storage scheduling method and system for a multi-service system, and belongs to the technical field of intelligent computing resource scheduling and storage management, and the method comprises the steps: collecting the real-time monitoring data of each node in the multi-service system, and carrying out the feature extraction; predicting a computing power demand value and a storage demand layering strategy in a future preset time window; dynamically correcting the prediction result, and generating a dynamic resource demand table and a cross-service priority weight matrix; generating a task allocation scheme through a hybrid scheduling algorithm; generating a storage data migration instruction through a storage scheduling engine; executing calculation node capacity expansion, task migration and storage data redistribution operation; and based on the executed latest state snapshot of the resource pool and the scheduling failure case library, updating a weight parameter of the time sequence prediction model and a scheduling strategy rule library to obtain an updated strategy version, and applying the updated strategy version to a next scheduling period. According to the invention, efficient scheduling of resources can be realized in a multi-service system environment.

The invention relates to the technical field of data processing, and discloses a multilevel cloud storage cluster construction and data storage method. The method comprises the steps that S1, multiple levels of storage layers are constructed and comprise a cache layer, a performance layer and a capacity layer, and the storage layers are provided with different storage media which are used for storing hot data, warm data and cold data respectively; s2, determining data popularity by monitoring data access frequency, access time interval and data importance; and S3, dynamically adjusting the storage hierarchy of the data according to the data popularity, and triggering hierarchical migration of the data. By constructing a multi-level storage layer and adopting a hierarchical architecture that the cache layer stores high-frequency data, the performance layer stores intermediate-frequency data and the capacity layer stores low-frequency data, high-heat data is preferentially stored in a high-performance medium, access delay is reduced, and data reading and writing efficiency is improved. Meanwhile, in combination with a dynamic data migration strategy, the storage position can be adjusted in real time according to the data access popularity, and the overall performance of the system is further optimized.

The embodiment of the invention discloses a data distributed storage method and device, and the method comprises the steps: carrying out the fragmentation processing of to-be-stored data, and obtaining an initial fragmentation result; utilizing a reinforcement learning model to perform popularity prediction on the data fragments according to the historical access data to obtain the access popularity of the data fragments; determining the number of copies based on the access popularity and importance of the data fragments, generating fragment copies of the data fragments according to the number of the copies, and storing the data fragments and the fragment copies into storage nodes according to the access popularity and node load state scores; when the loads of the storage nodes are unbalanced, data migration is carried out on the data fragments stored in the storage nodes, and when the node health state of a certain storage node is abnormal or the stored data fragments are lost, data reconstruction is carried out on the data fragments stored in the storage node. According to the method, the problems of non-uniform access load, resource waste, data loss and the like in a distributed storage system can be relieved.

The invention discloses a big data stream-oriented memory cache storage system, relates to the technical field of memory cache storage, and is used for solving the problem of poor memory cache storage management. According to the method, the access frequency, the timeliness and the service importance of the data are evaluated, the priority ranking model is constructed, the cache strategy is optimized based on the ranking result, the data storage and scheduling efficiency is improved, the write-in and read operation of the memory cache is adjusted in real time in combination with the flow prediction model, memory overload is effectively avoided, and the data storage and scheduling efficiency is improved. According to the method, non-emergency data writing is delayed by adopting a delayed writing and lazy loading strategy, data is loaded as required, meanwhile, a cache elimination mechanism is designed according to data priority, access frequency, survival time and size, data migration is carried out when a cache space is insufficient, reasonable resource allocation is ensured, and through a multi-level and dynamic cache management scheme, the cache management efficiency is improved. Therefore, the system performance, the resource utilization rate and the response speed are remarkably improved, the data processing efficiency is optimized, and cache conflicts are reduced.

The invention discloses a method and a system for realizing data synchronization between heterogeneous databases based on logs. The method comprises the following steps: acquiring connection information of a source database and a target database; querying a table structure of the source end database and generating an SQL statement to establish a data mapping rule; matching a corresponding rule model from a mapping rule model library; if the matching is not successful, the system abnormally quits and prompts the user to create a new mapping model and add the new mapping model to the mapping rule model base; converting a table structure of the source end database into a JSON character string based on a standard field type; an SQL statement required by the target database is output through the data mapping engine module; the message monitoring program reads the data in the temporary storage area and converts the data into data meeting the requirements of the target database; formatting the converted data into SQL statements supported by a target database; and sending the generated SQL statement to a target database to complete data migration. Accurate and controllable synchronization of the data and the model in a heterogeneous environment is realized.

The present invention relates to a data interaction method, device, equipment and storage medium of a multi-layer stacked memory, comprising the following steps: based on the access frequency clustering result, the target transmission data is mapped to a storage location to obtain a multi-layer storage allocation strategy table; based on the multi-layer storage allocation strategy table, the target transmission data is interactively planned to obtain an optimal interactive path plan; based on the optimal interactive path plan, the target transmission data is hierarchically migrated and scheduled to obtain a data migration execution queue; and the data migration execution queue is transmitted in parallel through an asynchronous pipeline transmission mechanism, which solves the technical problem that the existing data transmission strategies often lack in-depth analysis of the historical access characteristics of the storage hierarchy, resulting in unreasonable data distribution, and frequent cross-level data migration increases system delay and energy consumption.

The invention relates to the technical field of data storage, in particular to a dynamic data distribution storage method and system for an RAID (redundant array of independent disks), and the method comprises the following steps: based on the access frequency of data blocks in a disk, recording the access frequency and timestamp of each data block, carrying out heat rating on the data blocks, and determining thresholds of hot data and cold data. According to the method, the dynamic distribution of data storage is effectively optimized by analyzing the access frequency of the data blocks, and the performance and the economic efficiency of a storage system are remarkably improved by migrating the frequently accessed data to the cache disk and migrating the less accessed data to the low-cost storage; according to the method, unnecessary occupation of high-performance storage resources is reduced, the operation cost is also reduced, heat rating is performed on the data blocks, and the storage positions of the data blocks are correspondingly adjusted, so that the data processing speed is higher, meanwhile, the data security is improved, when a hard disk fails, data recovery is faster, and the service continuity is ensured.

A method for performing an application migration operation includes initiating the application migration operation to migrate an application from a source device to a target device, where the source device includes a source data migration agent and the target device comprises a target data migration agent. The method also includes initiating an active scan of network traffic at a transmission control protocol layer. The method further includes establishing a secure data path between the source device and the target device. In addition, the method includes beginning migration of the application from the source device to the target device using the secure data path. Moreover, the method includes making a first determination, using the active scan, that a data packet is associated with suspicious activity and rejecting the data packet based on the first determination.

The invention discloses a storage hard disk remote diagnosis system and method based on the Internet of Things, and relates to the technical field of health management of the Internet of Things and storage equipment. The system comprises a data fusion module, a causal analysis module, a risk analysis module and a map construction module. The data fusion module collects SMART parameters, IO operation time sequence data and environment data through Internet of Things equipment, dynamically distributes multi-source data weights by using an attention mechanism, and extracts hard disk health state features. And the causal analysis module is combined with the multi-scale time sequence convolutional network and the Bayesian causal network to identify periodic abnormal fluctuation and generate a fault root cause analysis report. The risk analysis module matches historical cases through federal incremental learning, calculates a hard disk fault risk index and generates an early warning signal. And the atlas construction module optimizes resource isolation, data migration and response paths according to the risk indexes, generates a visual operation and maintenance atlas, provides fault positioning, risk links and repair priorities, and improves the operation and maintenance management efficiency.

The invention provides a decision-making large model-oriented multi-level heterogeneous memory collaborative scheduling method, which comprises the following steps of: acquiring hardware configuration information and initial state information of a heterogeneous memory system, constructing a burst memory access model based on an Erlang distribution family, and acquiring dynamic memory access characteristics in combination with self-adaptive stratified sampling; representing the state of the memory system by adopting multi-dimensional efficiency measurement and a hierarchical affinity matrix; predicting load characteristics based on a generalized extreme value theory, and constructing an augmented Lagrange optimization model to generate a resource allocation scheme; formulating a data migration strategy through a Pareto optimization algorithm; and dynamic parameter adjustment of Thompson sampling and an exception handling mechanism based on a combined early warning index are introduced. According to the method, memory access features can be accurately described, resource allocation is optimized, migration overhead is reduced, and large decision model training efficiency is improved.

The present application relates to the field of storage, and discloses a data processing method and apparatus. The method is applied to an intelligent tiering system, the intelligent tiering system is used for carrying out intelligent tiering processing on service data in a storage system, and the storage system comprises at least one storage path. The method comprises: acquiring first service data, wherein the first service data is stored in a target storage path in the storage system, and the storage type of the first service data is an intelligent tiering type; acquiring an access log of the first service data; analyzing the access log of the first service data on the basis of a hot and cold identification algorithm configured by a user for the target storage path, to obtain the access popularity of the first service data; and sending a data migration instruction to the storage system on the basis of the access popularity of the first service data, so that a storage node of the storage system performs data migration on the first service data. The present application is used for reducing the impact on the storage system, improving the operation performance of the storage system.

The invention discloses a radioactive measurement data processing method for improving uranium exploration efficiency. The method comprises the following steps: standardized data acquisition: integrating a multi-parameter module, carrying out time-space synchronous acquisition of geological geophysical environment parameters, standardizing protocol alignment data, and supporting three-dimensional modeling; a three-dimensional coupling model is used for processing interference in a sub-module mode, LiDAR-DEM is used for correcting gamma attenuation in the terrain, an optical fiber thermopermeability instrument is used for correcting daughter errors in the hydrology, signal attenuation of a borehole is compensated through a drilling fluid chart, and a space-time continuous interference field is formed; performing dynamic equilibrium coefficient inversion: constructing equilibrium coefficient isoparametric mapping, and inputting rock core, logging and geochemical data into a random forest model; the transfer learning is trained by using historical data, and dynamic inversion and updating of a new area are carried out; intelligent data processing three-dimensional visualization: self-adaptive denoising and spectral shape matching reinforcement abnormity are carried out; constructing a three-dimensional model, and carrying out transfer learning to optimize the precision; the WebGL platform integrates multi-parameter display, virtual drilling functions and high-dimensional data intuitive interpretation.

The invention relates to the technical field of data processing, can be applied to business scenes of financial science and technology, medical treatment and health and the like, and discloses a data hierarchical storage method, device, equipment and medium based on access characterization. Writing the data into a matched first target storage device according to a preset device performance corresponding relation; after the data are stored, continuously monitoring subsequent access characteristics of the data, and dynamically judging whether the heat state is changed or not; and if the popularity state is changed and a specific migration condition is met, redistributing the second target storage device and completing data migration operation so as to realize automatic hierarchical storage management of cold and hot data. By acquiring the data access representation and dynamically identifying the popularity state, hierarchical migration of data with different popularity among storage devices with different performances is realized, the resource use efficiency and access performance of a storage system are improved, and the storage cost is reduced.

The invention discloses a laser processing parameter autonomous generation system and method based on digital twinning, and relates to the field of digital twinning, and the method comprises the steps: collecting and preprocessing processing data in real time through a multi-source sensor; processing and analyzing the task instruction by using a natural language, extracting a constraint condition and forming a structured demand; a processing parameter candidate set is generated through a Transform model in combination with historical data transfer learning, and virtual processing is performed by means of multi-physics field simulation; an improved non-dominated sorting genetic algorithm is adopted to dynamically optimize the parameter weight, and a global optimal parameter combination is obtained through digital twin iteration verification; and after full-process virtual processing verification and task demand comparison, parameters are adaptively corrected, and model parameters are continuously optimized according to physical and simulation data deviation after actual processing. The method has the advantages that the digital twin is used as a core, multi-source real-time data, the AI algorithm and multi-physical field simulation are fused, and autonomous generation, multi-target optimization and virtual-real closed-loop iteration of laser processing parameters are achieved.

The present application relates to a management method and management apparatus for a hybrid memory system, and a device and a medium. The method comprises: acquiring a write popularity value and an access popularity value of a currently accessed memory page; comparing the write popularity value with a specified write popularity threshold value, and generating write popularity page migration flag information; comparing the access popularity value with a specified access popularity threshold value, and generating access popularity page migration flag information; grouping hybrid memories in a hybrid memory system, and generating, on the basis of memory page information of all grouped memory banks, memory page management information corresponding to the hybrid memories; and on the basis of the write popularity page migration flag information, the access popularity page migration flag information, and current memory page management information corresponding to the currently accessed memory page, determining whether the current access memory page is a popular data page, and performing data migration according to a preset unpopular and popular data page migration strategy. The technical solution can improve the overall performance of a hybrid memory system having a horizontal architecture.

The invention discloses a market supervision data exchange platform migration and security enhancement method and system based on a credential environment, and relates to the technical field of network security. In the data migration process, the consistency and privacy protection of the data are ensured through Hash check and a multi-party security computing technology; a domestic commercial cryptographic algorithm and a quantum resistance encryption technology are introduced, so that the protection capability of the platform to existing security threats is effectively improved, and the resistance to future quantum computing threats is enhanced; an intrusion detection system and a virtual honeypot node are integrated, and a defense strategy is monitored and updated in real time in combination with a deep learning model, so that novel attack behaviors can be detected and coped with in real time; according to an intrusion detection alarm and a block chain audit result, a security policy and resource allocation are dynamically adjusted in combination with a machine learning technology, it is ensured that high-risk nodes are monitored in time, resource allocation is automatically optimized, and the resource utilization efficiency and security of the system are improved.

The invention discloses a distributed time sequence library data management method supporting cold and hot data level-to-level management, and relates to the technical field of computer databases, comprising: receiving a write-in request of time sequence data, and executing preliminary write-in data aggregation and sorting; dynamically identifying data cold and hot attributes based on a multi-dimensional data cold and hot degree calculation model in combination with a write-in behavior and an access behavior of time series data; according to the cold and hot attribute recognition result, in combination with the hierarchy boundary of self-adaptive division, automatic hierarchical storage of the time series data is executed; based on a cold and hot data dynamic migration and scheduling mechanism, according to the access frequency and time decay characteristics of the time series data, dynamically adjusting the storage hierarchy of the time series data, and executing a data migration task; and constructing a hierarchical index system adaptive to different cold and hot attribute time series data, and combining query frequency dynamic identification and index structure automatic upgrading. By adopting a cold and hot data automatic identification and hierarchical storage mechanism, the overall performance and the resource utilization rate of the system are improved.

The invention belongs to the field of big data, and relates to a data migration method, which comprises the following steps: obtaining storage cluster pressure test data, and constructing a performance model, the pressure test data being obtained by testing under different loads. And acquiring real-time monitoring data, and evaluating the current pressure state in combination with the performance model. And if the threshold value is exceeded, extracting a historical data change rule, and generating a load prediction model. And calculating a target migration concurrency number based on the real-time performance index and the prediction model. And adjusting the current data migration concurrence number and the bandwidth limiting parameter to target values. And finally, carrying out data migration of the storage cluster according to a preset protocol and the target parameter. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and the pressure measurement data and the real-time monitoring data can be stored in a block chain. The data migration efficiency can be improved.

The embodiment of the invention provides a data writing method and device of a storage system, a computer readable storage medium and electronic equipment. The method comprises the steps of obtaining historical data writing conditions of a plurality of logic address sections of the storage system; according to historical data writing conditions of the plurality of logical address sections, respectively determining writing heat of the plurality of logical address sections; and performing popularity grading on the to-be-written data according to the write popularity of the plurality of logic address sections to determine the popularity level of each to-be-written data. And writing the to-be-written data with the same heat level into the same physical address section of the storage system. According to the method, the to-be-written data with the same heat level is written into the same physical address section, classified writing of the data according to the heat level is realized, the heat of the data in the physical address section is relatively consistent, so that invalid data can be more effectively recovered due to the fact that the data with different heat is classified during data recovery, and the data recovery efficiency is improved. The data migration amount is reduced, and the recovery efficiency is improved.

The invention discloses a channel model dynamic association method based on a multi-source data index, and relates to channel engineering digital management. The channel model dynamic association method comprises the steps that component codes of channel engineering are constructed and used for identifying the same physical entity; establishing a three-layer index table structure according to the component codes; constructing a dynamic partition management mechanism of a three-level cache architecture, wherein the three-level cache architecture comprises a first-level cache stored in a memory, a second-level cache stored in a local disk and a third-level cache stored in a distributed manner; according to data access information in the three-layer index table structure, data migration is carried out among all levels of caches; according to the dynamic partition management mechanism, a data version management mechanism based on a timestamp concurrency control protocol is established, the data version management mechanism generates incremental data by comparing original data with updated data, and incremental updating or full updating is selected according to a threshold value. According to the method, the problem that data evolution tracking is difficult due to fragmentation of multi-source heterogeneous data and unbalanced access efficiency in navigation channel engineering is solved.

The invention discloses a data storage system of a digital country based on cloud computing, which comprises a client, a data transmission module and a cloud computing center, relates to the technical field of data storage, stores hot data on a high-performance hot data storage layer, ensures that frequently accessed data can be quickly responded, improves the system performance, and at the same time, stores the hot data on a high-performance hot data storage layer. The cold data is stored in a low-performance cold data storage layer, the storage cost is reduced, effective balance of performance and cost is achieved, the data is automatically migrated among different storage layers along with change of data access frequency, so that the performance and the cost are balanced, the frequently accessed data is migrated into a cache, the access speed of the data is increased, and the data access efficiency is improved. And establishing a relationship between a time interval and a data type of historical data, predicting data which are possibly accessed in a given time interval, adding the data into the cache, dynamically adjusting the cache, and ensuring the validity of the cache.

The invention provides a semantic-based migration and consistency verification method, system and device and a medium, and relates to the technical field of databases. The method comprises the following steps: performing metadata topology scanning analysis by obtaining metadata, including generating an abstract syntax tree and constructing a semantic graph; according to a metadata topology scanning analysis result, performing data mapping and conversion, including loading a YAML rule and generating corresponding type mapping and constraint conversion; data migration is carried out through primary key fragmentation parallel migration, batch writing optimization and real-time double-writing verification; through structure-data-business three-layer verification, simulation business SQL comparison and automatic difference repair, the integrity of migrated data and business compliance are ensured, the semantic gap problem in heterogeneous database migration is solved, high-precision and high-efficiency database migration is realized, the migration efficiency is improved, and the data migration efficiency is improved. The method is suitable for scenes with strict requirements on data consistency and migration efficiency, such as financial, government affair and enterprise-level data centers.

The invention provides a credential platform data migration verification method and system, and relates to the technical field of computers.The method comprises the steps that all to-be-migrated source data in a credential platform database are read, basic information of the source data is obtained, and a data label is set for each piece of data; reading fields and contents in the migration data, and constructing a data similarity verification model; selecting a migration path and a target database according to the migration demand, and dynamically evaluating the similarity between the source data and the target data through a data similarity verification model; verifying the migrated source data based on the target database, and performing incremental verification on data of different fields in the table; the data table of the to-be-migrated source data is verified in real time among different migration nodes, data changes on each node are verified step by step through an increment verification mechanism, if data missing or verification inconsistency occurs in the verification process, an alarm mechanism is triggered, and the problem of data missing or inconsistency can be automatically recognized and corrected.

Provided in the present disclosure are a task scheduling method applicable to a large language model (LLM), and a computing device, a storage medium and a computer program product. The method comprises: when determining that a target task processing node currently executing a target task meets a resource reporting condition, a target second task scheduler sending a target running resource of the target task processing node to a first task scheduler; when determining that the target running resource meets a task migration condition, the first task scheduler determining a migration task processing node from among a plurality of task processing nodes, and sending node information of the migration task processing node to the target second task scheduler; and the target second task scheduler determining the migration task processing node on the basis of the node information, determining the target task currently executed by the target task processing node, as well as task data of the target task, and migrating the target task and the task data in batches to the migration task processing node for execution, so as to obtain a task processing result of the target task.

The invention provides a marine engine lubricating oil system fault diagnosis method and system based on transfer learning from a simulation domain to a real domain, and the method comprises the steps: carrying out the fault simulation of different working conditions based on a built marine engine lubricating oil system simulation model, and obtaining a simulation fault signal response; acquiring real domain fault data of different working conditions of an engine lubricating oil system based on an engine experiment platform; a feature fusion structure based on an encoder-decoder is adopted to obtain simulation domain fault data; an engine lubricating oil system fault diagnosis model based on data migration and algorithm migration dual migration learning is obtained; and based on the engine lubricating oil fault diagnosis model, performing real-time fault diagnosis on the marine engine lubricating oil system to obtain a diagnosis result. According to the method, transfer learning from simulation domain fault features to real domain fault features is realized, so that the problems of high construction cost and insufficient generalization performance of a diagnosis model are solved.

The invention belongs to the field of data storage, and discloses a frequency access data storage migration method and device for big data, and the method comprises the steps: obtaining a data access log, carrying out the preprocessing of the data access log, and obtaining a time window statistical matrix; carrying out timeliness attenuation weighting calculation on a plurality of data access times in the time window statistical matrix to obtain a weighted access frequency vector; obtaining a multi-dimensional feature matrix based on the weighted access frequency vector; clustering the feature vector of each time window slice in the multi-dimensional feature matrix through a Gaussian mixture model to obtain a clustering tag vector; inputting the clustering label vector into an access frequency prediction model to obtain an access frequency prediction value of the data in a future preset time step; the migration decision matrix is determined based on the access frequency predicted value and the data storage cost parameter, data migration is carried out based on the migration decision matrix, the data migration complexity can be reduced, and the access speed and the storage cost are balanced.

The embodiment of the invention discloses a distributed migration method and device for a database and a medium, is applied to the technical field of data migration and is used for solving the problem that existing data migration is low in efficiency and unreliable. The method comprises the steps of collecting a DDL script of a to-be-migrated database object in a source end database, and performing grammar analysis on the DDL script to obtain a compatibility evaluation result of the current to-be-migrated database object; determining a data scale of the to-be-migrated database object based on a compatibility evaluation result, executing a task allocation strategy corresponding to the to-be-migrated database object according to the data scale, and allocating a migration task of the to-be-migrated database object to a task queue of a working cluster; and monitoring the working state when the working node executes the migration task of the task queue, and if the working node is determined to be abnormal based on the working state, determining the uncompleted migration task of the task queue based on the management node, and performing redistribution.

The invention discloses a memory data migration method and electronic equipment, and relates to the technical field of computers, and the method comprises the steps: determining the access heat of a storage device accepting the access of a computing node based on a memory access task executed by a switching controller and an address mapping table, so as to determine a source device and a candidate data unit, according to the access parameter of the candidate data unit and the state parameter of the storage device, calculating a performance score after the candidate data unit is stored in the storage device so as to determine the data unit to be migrated and the target device, so that cold and hot data identification from the perspective of a computer fast interconnection memory heterogeneous system is realized; the rationality of data migration is improved on the basis of quantitative performance scores; according to the method, the to-be-migrated data unit is controlled to be migrated from the source device to the target device, and the address mapping table is updated, so that the data migration task is prevented from influencing the host work of the computing node, and a cold and hot data migration scheme without being perceived by the host in the computer fast interconnection memory heterogeneous system is realized.

The invention discloses an intelligent water and fertilizer integrated irrigation system based on an Internet of Things sensor and a large language model, and relates to the technical field of intelligent agriculture. The system obtains soil, weather and plant growth state data in real time through Internet of Things sensors, including a soil sensor, a weather sensor and a multispectral sensor, deployed in a farmland; the method comprises the following steps: based on a large language model architecture, integrating preset agricultural field knowledge base data, performing field adaptive fine tuning on model parameters by adopting a data migration technology, and constructing an intelligent irrigation control model with a dynamic decision-making function; the control model analyzes sensor data and crop growth requirements, generates a water and fertilizer supply strategy, calls an internet-of-things control interface of the water and fertilizer all-in-one machine, and automatically adjusts irrigation water quantity and proportional supply of nutrient elements such as nitrogen, phosphorus and potassium. According to the invention, Internet of Things perception and large model decision are fused, and different environments are adapted through fine adjustment, so that closed-loop precise control is realized, and the water and fertilizer utilization rate and the crop yield are remarkably improved. Actual measurement shows that the system can improve the water and fertilizer utilization rate by 30% or above, and the crop yield is increased by 15-20%.

A method for performing an application migration operation, the method includes initiating the application migration operation to migrate an application from a source device to a target device, where the source device includes a source data migration agent and the target device includes a target data migration agent. The method also includes initiating an active scan. Further, the method includes establishing a secure data path between the source device and the target device. Moreover, the method includes making a first determination, using the active scan, that the network traffic or the source performance metrics are indicative of suspicious activity and logging information about the suspicious activity. The method also includes making a second determination that the application migration operation is complete and terminating the active scan and the secure data path based on the second determination.