157 results about "Data partitioning" patented technology

The invention provides an efficient query acceleration and cache optimization device for data knitting, which comprises a semantic partitioning component used for performing dynamic partitioning on a multi-source heterogeneous data set based on multi-source heterogeneous semantic metadata and embedding cross-domain association identifiers of data knitting to generate a semantic partitioning index table; the query intention mapping component is used for executing intention extraction on a query request initiated by a user to generate a query intention representation, and establishing a mapping relationship of the query intention representation to-be-queried target data partitions according to the semantic partition index table so as to generate a query-partition mapping table; the cache dynamic sorting component is used for generating a cache priority sorting table according to the query-partition mapping table and historical query frequency statistical data; and the scheduling and query execution component is used for optimizing the cache mechanism of the to-be-queried target data partition based on the cache priority sorting table and a preset proportion threshold value. According to the method, the time consumption of cache calling and source data repeated loading in the later query period is reduced.

The invention discloses a localization-adaptive track point visual rendering method based on partitioning and hierarchical caching, and the method comprises the steps: S1, inputting a track data stream, carrying out the reading according to the source type of the data stream, and carrying out the data partitioning and LOD processing of the track point data; s2, performing data analysis and rendering on the data; s3, carrying out batch processing on GPU vertexes, according to the state of the circular buffer, if the vertexes are writable, incrementally writing vertex data, and if Wrap is needed, carrying out loopback writing and updating a pointer; and S4, finally, the descriptors and the styles are updated, rendering increment drawing is carried out on the track points, and visual display is carried out. The method supports real-time rendering and playback of ten-million-level track points, has visual and instantiated rendering of window dynamic loading, space-time partitioning, hierarchical caching and WebWorker decoupling calculation, and improves the space-time data rendering capability of the domestic autonomous controllable field by ten thousand times; and the user experience is greatly improved by a non-perceptual interaction and transition smoothing algorithm.

The invention relates to the technical field of data cleaning, in particular to an automatic heterogeneous data cleaning method and system based on a configurable rule engine. The method comprises the following steps: acquiring a heterogeneous data source, uniformly converting a data mode and an instance into a multi-dimensional heterogeneous fact graph, calculating an execution weight for each conflict rule according to the authority of the data source and the priority of the rule when a write operation conflict is detected, segmenting data partitions by utilizing the betweenness centrality of a connected component and a node, the method comprises the following steps of: generating a plurality of data partitions, forming rule subsets corresponding to the data partitions, creating isolated Drools session instances to perform parallel cleaning, generating traceability logs in the parallel cleaning process, aggregating the traceability logs after all the data partitions are cleaned, and re-evaluating and updating the authority of each data source to solve rule conflicts in the next cleaning period. According to the scheme, the writing conflict can be precisely processed, the safety of parallel cleaning is improved, the processing period is shortened, and a continuously optimized processing closed loop is formed.

Various embodiments of the present disclosure provide a language model training technique. The language model training technique may include a data blending preprocessing step to improve the performance of the language model at an enterprise level. The data blending technique includes receiving an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source, receiving a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source, storing the enterprise data partition as an initial training partition of a plurality of balanced training partitions within a balanced training dataset, and generating a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition. A domain-specific language model may then be trained based on the balanced training dataset.

The invention provides a master-slave device write operation communication method and communication system, and the method comprises the steps: a master device sends a shared address corresponding to a current write operation, so that a slave device confirms whether the current write operation is matched or not; and the master device sends a write operation command, the identifier set, the write address and the write data in sequence, so that the target slave devices whose own identifiers belong to the identifier set obtain the target data according to the identifier arrangement sequence in the identifier set and the target data length of each target slave device, and data write is completed. A plurality of slave devices share one address and are distinguished in combination with identifiers, so that address resource waste is avoided, and the upper limit of the number of deployable slave devices is improved. In the single I2C transaction, based on the identifier arrangement sequence and the target data length, write data partitioning processing is realized, the occupation and delay of a bus are reduced, and the communication efficiency is improved. The participation sequence and any combination of the slave devices can be dynamically specified, additional mechanisms such as mapping tables are not needed, and the hardware complexity and power consumption of the slave devices are prevented from being increased.

The present application relates to the technical field of storage, and discloses a data attribute-aware storage system and data management method for a key-value database. The method comprises two parts, i.e., data hotness attribute-based writing and data density attribute-based merging. The data hotness attribute-based writing comprises classifying key-value pairs into cold key-value pairs and hot key-value pairs, to realize separate storage of cold data and hot data by means of a data hotness attribute-based writing strategy. The data density attribute-based merging comprises: when key-value pairs in an SSTable are extracted during a merging operation and form a sorted key-value pair sequence, extracting the key-value pairs one by one, and calculating a binary difference between every two adjacent key-value pairs; and if the difference is greater than a set threshold, stopping filling the current SSTable, and creating a new SSTable. In the present invention, by analyzing data hotness and data density, data having different attributes are classified to different data partitions, thereby reducing I / O amplification caused by repeated reading and writing in a compression process.

The invention belongs to the technical field of SaaS platforms in the online car-hailing industry, and particularly relates to an online car-hailing data configurable auditing method based on a SaaS multi-tenant architecture, which comprises the following steps: acquiring online car-hailing data of multiple tenants; based on a multi-tenant data model of logic isolation, data partitioning is carried out through tenant identifiers, data security isolation and high-performance access among tenants are carried out in combination with a horizontal extension architecture of sub-libraries and sub-tables, exclusive isolated data of the tenants are obtained, corresponding auditing rules are loaded through a unified rule engine, and data security isolation and high-performance access are carried out. Generating a structured executable auditing rule set corresponding to the current tenant; inputting an AI large model to carry out intelligent identification and feature extraction, outputting a preliminary auditing result, and carrying out logic verification and confidence fusion through a tenant exclusive rule and a rule engine to generate a final auditing state; and updating the data state library, and triggering an alarm or notification mechanism. Therefore, the problems that in the prior art, a data isolation mechanism is weak, auditing rule configuration is rigid, and auditing capacity is single are solved.

A line replaceable unit (LRU) includes a data communication interface configured to receive identification (ID) data from an aircraft communication bus, an aircraft systems interface configured to receive aircraft systems data corresponding to an operation of at least one aircraft system, and an LRU data interface configured to receive LRU data corresponding to the LRU. The LRU further includes a processor and a memory unit. The processor processes the aircraft systems data and the LRU data and determine at least one data category. The memory unit includes a plurality of data partitions. The processor is configured to generate at least one configuration data package by combining the ID data with one or both of the aircraft systems data and the LRU data, and to store the configuration data package in a given data partition based on the data category.

The invention discloses a data aggregation processing method and device, computer equipment and a storage medium, belongs to the technical field of big data, and is applied to data aggregation in the financial field. The method comprises the following steps: firstly, grouping dimension fields by utilizing a mapping rule and a hierarchical clustering algorithm; and secondly, dynamic grouping mapping is established, one-time scanning is performed on the source data table, and each data record is directly attributed to a corresponding grouping structure, so that the data processing efficiency is improved. Meanwhile, a correlation index of the index and the data partition is constructed based on a Hash mapping algorithm. Under the support of an indexed data structure, the aggregation calculation function can efficiently execute operations such as summation and averaging, and rapid aggregation of multi-dimensional data is realized. Generally speaking, according to the method, the correctness of an aggregation result is ensured, meanwhile, the calculation and storage pressure of the system is remarkably reduced, the timeliness and expandability of data analysis are improved, and the method is particularly suitable for real-time or quasi-real-time multi-dimensional analysis requirements in a large-scale business scene.

The invention relates to a data recovery method and device, equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that in response to the fact that a terminal is in a first operating system boot starting state, target data in a first data partition is determined; generating a block device mapping graph of the target data, and storing the block device mapping graph to the target data partition; in response to the condition that the terminal is in a factory setting restoring state, acquiring a block device mapping graph from the target data partition by using a second operating system; reading the target data from the block device where the first data partition is located into a memory according to the block device mapping graph; and recovering the target data to the first data partition by using the second operating system. According to the method, the block device mapping graph is stored in the target data partition, so that the second operating system can reserve the target data based on the block device mapping graph, important data are reserved and recovered when factory settings of the terminal are recovered, and it is guaranteed that the data are available and not lost.

This disclosure is directed to systems and methods of data partitioning in image encoding. A video encoder may receive a single stream of video data that includes multiple layers. The video encoder may encode the one or more layers utilizing multiple counters for each component within the layers. The multiple counters may correspond to the header bits, luma bits, and chroma bits within each slice layer. The encoded layers may then be assembled into a single slice before it is sent to a decoder for decoding of image frame.

This invention provides a data storage method and system for tuberculosis detection, relating to the field of tuberculosis detection technology. The data storage method for tuberculosis detection includes the following steps: S1. Collecting whole-genome sequencing data of Mycobacterium tuberculosis, host serum IgG titer, and drug sensitivity test results; S2. Calculating genetic distance D based on a reverse evolution model to generate four-dimensional spatiotemporal coordinates (t, x, y); S3. Performing data partitioning and storage based on the drug target barrier value β; S4. Generating dynamic metadata using a host-pathogen dynamics model and compressing and storing it. This invention implements a dynamic storage entropy adjustment algorithm at the hardware and software collaborative level, continuously optimizing the matching efficiency of data compression and physical storage. This results in an intelligent data hub that can perceive the evolutionary pulse of pathogens and autonomously optimize resources, providing support for clinical tuberculosis prevention and control decisions with temporal depth, spatial correlation, and risk evolution.

The invention discloses a file storage processing method and system based on artificial intelligence, and the method comprises the steps: obtaining access description data of a file entity in a file storage data set, and the access description data comprises an access activeness parameter and access result data; mapping the data partition to a storage unit of a corresponding activeness interval based on the activeness parameter; controlling each storage unit to construct access sequence structure information containing sequence and non-sequence arrangement description; generating active and inactive file corresponding relation result data based on the information; evaluating the access change of the target scene file in combination with the multi-interval result to obtain an access change measurement result; therefore, the storage structure is adjusted, the storage resource configuration is matched with the scene file access change characteristics, and the storage resource utilization rate and the access performance are improved.

This invention belongs to the field of soft measurement modeling technology and discloses a soft measurement modeling method based on mode-related spatiotemporal diffusion generation, which includes the following steps: (1) acquiring data of multi-mode dynamic processes; (2) data partitioning and preprocessing operations; (3) establishing a mode-related spatiotemporal diffusion model and generating virtual samples; (4) predicting pressure variables of three-phase flow processes and evaluating model performance. This invention proposes a soft measurement modeling method based on mode-related spatiotemporal diffusion generation. By using the DSTN-Net noise prediction network to capture the dependence of dynamic data in the time and space dimensions, the mode learner learns the multi-mode distribution characteristics and distinguishes the data distribution of different modes, which can generate virtual samples with high similarity to the original samples, thereby improving the prediction performance of the soft measurement model in the case of small samples.

This invention discloses a method and system for automatically creating table partitions within a dynamic range of MySQL tables, belonging to the field of database technology. The technical problem it addresses is how to solve the inefficiency of manually maintaining table partitions, achieving dynamic partition management for newly added tables, and reducing operational burden and human error risks. The method includes the following steps: Table collection: Dynamically obtain a list of table names requiring partition maintenance from a specified MySQL database using a Linux shell script, and use the tables in the list as target tables; Table partition management: For each target table, create new partitions according to a time period using a Linux shell script, and delete historical data partitions; Partition maintenance: Achieve fully automated partition maintenance using a Linux crontab scheduled script.

This invention discloses a method, device, and storage medium for secure data backup and migration of storage devices, relating to the field of data backup technology. The method includes the following steps: dividing file data in the storage device using a file data partitioning method to obtain a set of file data blocks; obtaining feature parameters of all file data blocks in the set and obtaining feature vectors for each file data block, denoted as file data block feature information; constructing a data block filtering feature vector and building a document feature filtering table based on the file data block feature information, and performing an initial backup; and performing a second backup based on the file data partitioning method and the document feature filtering table. This invention addresses the problem that existing data backup technologies, when performing incremental backups of file data, cannot simultaneously guarantee backup efficiency and data reliability while avoiding the need to restore each backup individually during data recovery.

The invention relates to the technical field of data visualization, in particular to a security visual monitoring method based on big data. The method comprises the following steps that original data streams of a plurality of service systems are obtained and classified according to source types, a data partition set is formed, and the data partition set comprises a plurality of data partitions; then extracting basic attributes of each data block to generate a data activity feature table, constructing a data sensitivity visual mapping unit, and converting feature weights into a safe visual mapping graph; monitoring an intensity change abnormal area in the mapping graph, and determining a safety risk area; and generating risk warning information according to the risk area, and outputting the risk warning information to the safety visual monitoring terminal to realize real-time risk identification and early warning. Through multi-source data fusion modeling, sensitivity visual mapping, adaptive anomaly aggregation detection and a risk feedback closed loop mechanism, real-time visual perception, accurate identification and dynamic management and control of security risks in a big data environment are realized.

The invention discloses a distributed file system oriented to consumer-level equipment and a data security management method of the distributed file system, and belongs to the technical field of computer systems. The system comprises a virtual namespace management module used for maintaining a virtual namespace tree formed by combining a local main namespace tree and a remote snapshot sub-tree; the access control and encryption module is used for executing access control based on the attribute set of the virtual namespace tree nodes; and the data processing pipeline module is used for partitioning the file and realizing pipeline operation of transmission and decryption through independent data flow and control flow channels. According to the invention, the access control and the encryption strategy are tightly coupled based on the unified attribute set, so that file re-encryption is not needed when the authority is changed, and the system flexibility is improved; and the security operation delay is hidden through data partitioning and pipeline processing, so that the file access efficiency is improved, and the problem of balance among security, flexibility and high efficiency of the distributed file system of the consumption equipment is effectively solved.

The application relates to the technical field of medical data privacy protection, and discloses a surgical operation data privacy protection system based on a block chain. After a surgical operation data classification module obtains surgical operation data, the data content is analyzed, and data partitions are divided based on data attributes; a privacy policy mapping module obtains the data partitions, associates privacy policies stored in a block chain, and establishes a mapping relationship between the data partitions and the privacy policies; a key combination module extracts key partitions based on the mapping relationship, combines encryption keys, and calculates key matching degrees; a policy updating module updates the privacy policies and adjusts the mapping relationship based on the key combination and access records; and an access management module processes access requests and generates an access control library based on the updated mapping relationship. Through the collaborative operation of the multiple modules, the adaptability and reliability of surgical operation data privacy protection are improved.

The application discloses a graph data partitioning method based on a cache mechanism, comprising the following steps: obtaining graph data, obtaining the total number of graph vertices, the neighbor list of each vertex, the initial vertex partition set and the initial partition number according to the graph data; calculating the upper limit of the load; calculating the correlation score of the vertex in each vertex set which does not reach the upper limit of the load according to the vertex and the neighbor list of the vertex; judging whether the vertex meets the cache condition, if yes, partitioning according to the correlation score; if not, the vertex is partitioned into the cache module, when the cache module reaches the maximum capacity limit, the cache module returns the vertex and the neighbor list which are most suitable for the current partitioning, and the process returns; judging whether all vertices have been partitioned, if yes, the final graph data partition set is obtained; if not, the process returns. The application improves the partitioning precision of the Fennel algorithm under the condition that the size of the graph data is too large.

Provided in the embodiments of the present disclosure are a hash join processing method, system and device, and a storage medium and a program product. The method comprises: scanning a plurality of rows of data in a first data table within a target scanning range corresponding to a target worker thread, and on the basis of a first hash join mode, writing the plurality of rows of data into a first hash table in a memory; if it is determined that an available memory capacity corresponding to a hash join task is lower than a set threshold, sending the scanned plurality of rows of data to a data partitioning management thread, and switching to use a second hash join mode; and receiving first target row data distributed by the data partitioning management thread, and on the basis of the second hash join mode, writing the first target row data into a second hash table corresponding to the target worker thread in the memory, so as to perform persistence processing of the data in the second hash table. The present disclosure can flexibly switch between different hash join modes on the basis of the memory occupancy of a data table for hash table creation.

The invention discloses a cache management method and device, equipment, a storage medium and a product, and the method comprises the steps: responding to a cache access, and obtaining a hit condition of the cache access; adjusting global leapfrogging parameters according to the hit condition; and according to the hit condition and the global leapfrogging parameter, adjusting the item level of the data item. According to the embodiment of the invention, the adjustment step length of the item level of the data item is reduced during hit, it is ensured that the hot data partition stores the high-frequency accessed data item, the occasionally accessed data item is prevented from mistakenly entering the hot area, the hot data partition entering speed of the sudden hot data item can be reduced, and the hot data partition entering efficiency is improved. The concerned data items are prevented from being quickly deleted; when the new inserted data is not hit, a large entry level is configured for the new inserted data, the new inserted data is placed in a cold data area, storage of high-frequency access data is prevented from being affected, negative feedback closed-loop control over a cache area is achieved, cold and hot data are effectively distinguished, the problem of access delay caused by the fact that the cold and hot data are difficult to distinguish is solved, and the access efficiency is improved. And the response speed and the cache stability are dynamically balanced.

The invention relates to a localized database migration and safe operation and maintenance method and system for a thermal power system, and belongs to the technical field of databases. The method comprises the steps that an existing database of the thermal power system is comprehensively evaluated, and a migration plan is made; selecting and building a domestic database platform supporting a hybrid storage engine, wherein the platform respectively processes high-frequency instructions, historical data and equipment metadata by utilizing a memory optimization table, a time sequence database and a relational database; data migration is executed in a planned power failure window or a mirror image test environment, and consistency verification, business function verification and performance pressure test are carried out; executing data governance, including measurement point naming standardization, abnormal data cleaning and data partitioning; after migration is completed, a whole-process security protection system is deployed, transmission is encrypted by adopting a national cryptographic algorithm, and strict access control and two-factor authentication based on roles are implemented. According to the method, the system performance, the data quality and the information security protection level are remarkably improved, and the operation and maintenance cost and complexity are reduced.

The application relates to the field of computer technology and artificial intelligence, and proposes an IDE scheduling platform task monitoring method and system based on a large model, wherein the method comprises the following steps: collecting task running data, system performance data and task running log text data from an IDE scheduling platform in real time; extracting key feature task dependency relationships, resource occupation trends and data partition size variances as a task feature matrix; inputting the task feature matrix into a Transformer large model to output the timeout probability of the corresponding task; triggering an early warning when the timeout probability exceeds a preset timeout threshold; fusing abnormal performance data in the task system performance data and fault keywords in the task running log text data to obtain an abnormal task feature matrix; inputting the abnormal task feature matrix into the Transformer large model to output a fault root cause by using an attention weight mechanism. The application solves the problems of low task monitoring efficiency, insufficient precision and resource waste of the IDE scheduling platform.

Based on the next write address, the relevant partition is selectively written directly to the volatile memory (e.g., SRAM) of the controller and the remaining partition is written to the volatile memory (e.g., SRAM) of the host. A controller (e.g., a DRAM or HMB) avoids the need to make any additional read from the host's volatile memory and any additional write from the controller's volatile memory. The need to avoid any additional read from the host's volatile memory, which has slower access times than the controller's volatile memory, improves exit latency in boot and low power state exit procedures. The controller may store the parity data partition and / or the next write address in the non-volatile memory prior to writing the parity data partition to the controller or the host. The next write address is then evaluated to determine whether the parity data partition is written to the volatile memory of the controller or the volatile memory of the host.

The application provides a digital elevation data service method and device. The method comprises the following steps: obtaining global digital elevation data to be stored, determining a data storage structure, a data storage path and a data partitioning rule corresponding to the global digital elevation data; performing hierarchical recursive partitioning processing on the global digital elevation data in a parallel mode to obtain tile data to be stored of a slice; storing the tile data to be stored of the slice into a tile database based on the data storage structure, the data storage path and the data partitioning rule; obtaining a tile data acquisition request, determining a target level of the tile data acquisition request, and returning tile data of a corresponding level based on a relationship between the target level and a preset level. The method provided by the application can improve the standardization of digital elevation data storage, shorten the service response time of digital elevation data request, and effectively improve the service response efficiency of digital elevation data.

This invention discloses a flash memory management method for edge computing devices. The invention relates to the field of software engineering technology and includes the following steps: flash memory selection, root device partitioning, dividing the flash memory device into different data partitions, data compression, wear management (including heat calculation, wear threshold calculation, and migration priority calculation), migrating data to other underutilized flash memory blocks, determining the data migration order based on migration priority P, and data backup. By calculating the wear threshold of a flash memory block and comparing it with the heat value of the flash memory block, data is migrated to other underutilized flash memory blocks, thereby ensuring balanced use of each flash memory block, slowing down wear, and extending lifespan. Simultaneously, by calculating migration priority during data migration, critical data is ensured to be migrated first, improving the performance and stability of the edge computing device system.

The invention discloses a shared memory and load balancing design based on database agents. The dynamic partitioning and transaction scheduling scheme based on the PostgreSQL shared cache database, which is discussed emphatically, has extremely important research value in the aspects of quickly solving transaction conflicts, reducing page transfer quantity in transactions, relieving load imbalance and the like. The distributed shared cache architecture naturally has the capability of automatically and dynamically partitioning data, the cost of data transfer is extremely low, and manual intervention is not needed. Therefore, conflicts between concurrent transactions and a frequent page transfer phenomenon between nodes can be reduced through a mode of combining data partitioning and transaction scheduling. Due to the fact that the cost of page transfer between the computing nodes is low, when cross-partition transactions are continuously increased, only a small amount of page transfer operation needs to be carried out, and distributed transactions do not need to be processed. When the load changes, the data can be quickly subjected to re-partitioning operation by means of a load awareness technology and by adjusting the logic of transaction scheduling.

The application discloses a transaction concurrent control method, first constructs a set of transaction system, each transaction index in the transaction system collects the transaction characteristic information of the data item of the partition in real time and dynamically adjusts the partition of the transaction, selects the appropriate concurrent control method for each partition, generates a concurrent control plan, and gives the partition manager, the partition manager adjusts the data partition in the transaction index according to the concurrent control plan, and distributes the concurrent control plan to the client, according to the partition range in the concurrent control plan and the concurrent control method of each partition, carries out the read-write operation, and completes the adaptive transaction concurrent control method. The method of the application adopts the adaptive concurrent control mechanism based on the transaction characteristics of the data item, dynamically selects the appropriate concurrent control method according to the transaction scale, the conflict rate and other characteristics, balances the load between the nodes, and effectively deals with the real-time variable transaction type and different workloads.