280 results about "Data partitioning" patented technology

The invention relates to an experimental data processing method and device, an AI analysis module and computer equipment, and belongs to the field of data processing.The method comprises the steps that multi-dimensional original data are partitioned according to types, and formats are unified; generating a similarity matrix based on time and space neighborhood information, and marking abnormal fluctuation points; effective signals are separated through a time-frequency feature matching noise library; extracting multi-layer features of basic statistics, time sequence correlation and trend change; and dynamically screening core features to update the tracking type experimental model. The matched AI analysis module integrates hardware circuits of data partitioning, similarity calculation, anomaly marking, noise matching, feature extraction and model updating, and whole-process acceleration is achieved. According to the method, through multi-dimensional data compatibility processing, accurate anomaly detection, multilayer feature fusion and model adaptive optimization, the experimental data processing efficiency and conclusion reliability are remarkably improved, and the method is suitable for real-time analysis of multiple scenes such as scientific research and industry.

The invention relates to a firmware upgrading method and electronic equipment. The method comprises the following steps: in a first stage of upgrading, constructing upgrading meta-information according to a current partition table and a to-be-upgraded firmware data storage area base address obtained by calculating to-be-upgraded firmware meta-information, and according to a block list description file, updating the to-be-upgraded firmware data storage area base address; in the first-stage upgrade, a partition table file with a preset name in a firmware upgrade package under a preset firmware upgrade package path of an encrypted preset user data partition and all to-be-upgraded firmware mirror image files are stored in an address space determined by a base address of a to-be-upgraded firmware data storage area; and determining a to-be-upgraded firmware data storage space end address and a to-be-upgraded firmware data moving base address according to the guide communication partition address space change mark, and copying data in an address space determined by the to-be-upgraded firmware data storage space end address to the to-be-upgraded firmware data moving base address. According to the invention, under the condition of user data partition encryption, the firmware with any change of the partition table can be safely and reliably upgraded.

The invention relates to the technical field of big data processing, and particularly discloses a computer data management system based on big data. The system comprises a distributed data acquisition module which captures multi-source data through an API gateway and a crawler and embeds a quality label; the intelligent storage scheduling engine is used for realizing automatic migration of cold and hot data by adopting a column-type and distributed hybrid architecture; fusing a computing framework, and dynamically coordinating flow processing and batch processing resource execution feature extraction; the self-adaptive strategy center is used for generating data partitions and encryption strategies based on reinforcement learning; and a multi-level security protection system is adopted, and attribute base decryption and dynamic desensitization are deployed. The problems of resource scheduling rigidity and security protection lag in the prior art are solved, the storage cost is reduced by more than 40%, the query delay is compressed to be within 200ms, and the method is suitable for real-time decision-making scenes in the fields of e-commerce and finance.

The invention discloses a cable defect analysis method and system in a cable laying process, and the method comprises the steps: dividing a current data sequence according to a preset data division strategy, obtaining at least one current data sub-sequence, and obtaining cable images of different monitoring positions according to the at least one current data sub-sequence; preprocessing the cable image to obtain a target cable image, and performing cable contour and background region segmentation on the target cable image to obtain a cable contour image; calculating a cable area based on the cable contour image, and extracting cable surface features; analyzing the surface features of the cable based on a set noise threshold, and removing noise features to obtain optimized surface features of the cable; and analyzing the surface features of the optimized cable based on the defect detection model, and outputting surface defect information. The accuracy and the real-time performance of defect detection are remarkably improved, the manual inspection cost and the leakage detection risk are reduced, and reliable guarantee is provided for safe operation of the cable.

The invention discloses a database data storage system based on multi-level cache, and relates to the technical field of database data storage systems, the database data storage system comprises a basic storage layer, a first-level cache layer, a second-level cache layer, a cache cooperative control module and a data read-write scheduling module, the basic storage layer is used for persistently storing total data of a database; the first-level cache layer is divided into a plurality of hot data partitions; and the second-level cache layer is provided with quasi hot data partitions in one-to-one correspondence with the hot data partitions. According to the database data storage system based on the multi-level cache, the system constructs a'high frequency-secondary high frequency-full quantity 'layered architecture through the first-level cache, the second-level cache and the basic storage layer, dynamic flow of data is achieved in cooperation with a data aging counter and a heat recovery detector, heat reduction data is migrated from the first-level cache to the second-level cache, and the heat reduction data is transferred to the second-level cache. And the heat recovery data returns to the first-level cache from the second-level cache to ensure that the high-frequency data is always kept in a cache layer.

The invention provides an efficient query acceleration and cache optimization device for data knitting, which comprises a semantic partitioning component used for performing dynamic partitioning on a multi-source heterogeneous data set based on multi-source heterogeneous semantic metadata and embedding cross-domain association identifiers of data knitting to generate a semantic partitioning index table; the query intention mapping component is used for executing intention extraction on a query request initiated by a user to generate a query intention representation, and establishing a mapping relationship of the query intention representation to-be-queried target data partitions according to the semantic partition index table so as to generate a query-partition mapping table; the cache dynamic sorting component is used for generating a cache priority sorting table according to the query-partition mapping table and historical query frequency statistical data; and the scheduling and query execution component is used for optimizing the cache mechanism of the to-be-queried target data partition based on the cache priority sorting table and a preset proportion threshold value. According to the method, the time consumption of cache calling and source data repeated loading in the later query period is reduced.

A method and a system for utilizing dynamic data partitioning to ensure consistency of event publications with associated business transactions in a distributed microservices architecture in order to optimize efficiency in polling data are provided. The method includes: defining a hash space that includes a range of assignable hash values; deploying a respective instance of each microservice to form a cluster of microservices within the distributed microservices architecture; allocating a respective subset of the hash space to each microservice; and facilitating a data polling capability with respect to a data table based on the allocated respective subset of the hash space and the deployed respective instance for each microservice.

The invention discloses an aerospace equipment performance evaluation method based on digital test data fusion, and belongs to the field of electronic engineering and computer science, and the method comprises the steps: designing a physical test and a digital test, selecting related indexes of aerospace equipment, and carrying out corresponding physical and digital tests around the related indexes of the aerospace equipment; performing preprocessing and multi-source data integration on the physical test data; performing data quality evaluation and virtual data division on the digital test data; and performing double fusion on the result through an adaptive fusion algorithm to obtain a performance evaluation result. The advantages of the digital test and the physical test can be fully exerted, and comprehensive and accurate evaluation of the aerospace equipment performance is realized through deep fusion of the digital test data and the physical test data.

The invention discloses a localization-adaptive track point visual rendering method based on partitioning and hierarchical caching, and the method comprises the steps: S1, inputting a track data stream, carrying out the reading according to the source type of the data stream, and carrying out the data partitioning and LOD processing of the track point data; s2, performing data analysis and rendering on the data; s3, carrying out batch processing on GPU vertexes, according to the state of the circular buffer, if the vertexes are writable, incrementally writing vertex data, and if Wrap is needed, carrying out loopback writing and updating a pointer; and S4, finally, the descriptors and the styles are updated, rendering increment drawing is carried out on the track points, and visual display is carried out. The method supports real-time rendering and playback of ten-million-level track points, has visual and instantiated rendering of window dynamic loading, space-time partitioning, hierarchical caching and WebWorker decoupling calculation, and improves the space-time data rendering capability of the domestic autonomous controllable field by ten thousand times; and the user experience is greatly improved by a non-perceptual interaction and transition smoothing algorithm.

The invention relates to the technical field of data cleaning, in particular to an automatic heterogeneous data cleaning method and system based on a configurable rule engine. The method comprises the following steps: acquiring a heterogeneous data source, uniformly converting a data mode and an instance into a multi-dimensional heterogeneous fact graph, calculating an execution weight for each conflict rule according to the authority of the data source and the priority of the rule when a write operation conflict is detected, segmenting data partitions by utilizing the betweenness centrality of a connected component and a node, the method comprises the following steps of: generating a plurality of data partitions, forming rule subsets corresponding to the data partitions, creating isolated Drools session instances to perform parallel cleaning, generating traceability logs in the parallel cleaning process, aggregating the traceability logs after all the data partitions are cleaned, and re-evaluating and updating the authority of each data source to solve rule conflicts in the next cleaning period. According to the scheme, the writing conflict can be precisely processed, the safety of parallel cleaning is improved, the processing period is shortened, and a continuously optimized processing closed loop is formed.

A method and system for enhanced financial data processing using large language models (LLMs) to transform PDF statements into structured accounting data. The invention preprocesses statements with line numbering, character position tagging, and data partitioning to overcome LLM context limitations. The system extracts transactions, reconciles them with statement balances, normalizes vendor information, and generates enhanced output files compatible with standard accounting platforms. This enhanced transaction data collapses reconciliation and categorization times in downstream accounting systems. The system creates a robust vendor-transaction database supporting direct analysis or simultaneous application of multiple accounting models.

The invention relates to the technical field of data processing, and particularly discloses a Tibetan word segmentation method and system based on a multi-language pre-training model CINO, and the method comprises the steps: collecting a to-be-labeled data set, and providing massive text resources for subsequent research; then word segmentation conversion is carried out to obtain a to-be-trained data set, the text is converted into a lexical element sequence suitable for model processing, a model learning structure is facilitated, attribute parameters of the to-be-trained data set are analyzed, whether data division is carried out or not is judged accordingly, and reasonable division can guarantee representativeness of a training set and a verification set, avoid data distribution deviation and improve the generalization ability of the model; then a training and verification data set is obtained through division and used for training a multi-language pre-training model CINO, training process parameters are collected and analyzed, the model training state can be informed, strategies and hyper-parameters can be adjusted in time, and model initialization is completed, so that Tibetan word segmentation accuracy and reliability are promoted to be improved, and application of the multi-language processing technology in the Tibetan field is assisted.

Various embodiments of the present disclosure provide a language model training technique. The language model training technique may include a data blending preprocessing step to improve the performance of the language model at an enterprise level. The data blending technique includes receiving an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source, receiving a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source, storing the enterprise data partition as an initial training partition of a plurality of balanced training partitions within a balanced training dataset, and generating a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition. A domain-specific language model may then be trained based on the balanced training dataset.

The invention provides a master-slave device write operation communication method and communication system, and the method comprises the steps: a master device sends a shared address corresponding to a current write operation, so that a slave device confirms whether the current write operation is matched or not; and the master device sends a write operation command, the identifier set, the write address and the write data in sequence, so that the target slave devices whose own identifiers belong to the identifier set obtain the target data according to the identifier arrangement sequence in the identifier set and the target data length of each target slave device, and data write is completed. A plurality of slave devices share one address and are distinguished in combination with identifiers, so that address resource waste is avoided, and the upper limit of the number of deployable slave devices is improved. In the single I2C transaction, based on the identifier arrangement sequence and the target data length, write data partitioning processing is realized, the occupation and delay of a bus are reduced, and the communication efficiency is improved. The participation sequence and any combination of the slave devices can be dynamically specified, additional mechanisms such as mapping tables are not needed, and the hardware complexity and power consumption of the slave devices are prevented from being increased.

The invention discloses a compilation optimization method and device for accelerating Attention calculation, and belongs to the field of compilation optimizing.The method comprises the following steps that multiple scheduling strategies are generated according to the number, shape and Attention type of input matrixes and a target hardware platform; defining a cost function, evaluating time and space overhead of each scheduling strategy, and selecting an optimal strategy based on hardware platform characteristics and user requirements; the method comprises the following steps of: performing multi-core parallel division on input data in sequence length and Head dimension, and reducing delay through asynchronous communication overlapping calculation and communication; in a training stage, a random floating point matrix generated by hardware and an attention weight are used for carrying out Hadamard product operation, and a Dropout function is realized. According to the method, the characteristics of different hardware platforms can be flexibly adapted through the generated Attention operator template, excellent performance can be achieved under various shapes, and efficient data division is achieved by optimizing balance between calculation and communication.

The present application relates to the technical field of storage, and discloses a data attribute-aware storage system and data management method for a key-value database. The method comprises two parts, i.e., data hotness attribute-based writing and data density attribute-based merging. The data hotness attribute-based writing comprises classifying key-value pairs into cold key-value pairs and hot key-value pairs, to realize separate storage of cold data and hot data by means of a data hotness attribute-based writing strategy. The data density attribute-based merging comprises: when key-value pairs in an SSTable are extracted during a merging operation and form a sorted key-value pair sequence, extracting the key-value pairs one by one, and calculating a binary difference between every two adjacent key-value pairs; and if the difference is greater than a set threshold, stopping filling the current SSTable, and creating a new SSTable. In the present invention, by analyzing data hotness and data density, data having different attributes are classified to different data partitions, thereby reducing I / O amplification caused by repeated reading and writing in a compression process.

The invention belongs to the technical field of SaaS platforms in the online car-hailing industry, and particularly relates to an online car-hailing data configurable auditing method based on a SaaS multi-tenant architecture, which comprises the following steps: acquiring online car-hailing data of multiple tenants; based on a multi-tenant data model of logic isolation, data partitioning is carried out through tenant identifiers, data security isolation and high-performance access among tenants are carried out in combination with a horizontal extension architecture of sub-libraries and sub-tables, exclusive isolated data of the tenants are obtained, corresponding auditing rules are loaded through a unified rule engine, and data security isolation and high-performance access are carried out. Generating a structured executable auditing rule set corresponding to the current tenant; inputting an AI large model to carry out intelligent identification and feature extraction, outputting a preliminary auditing result, and carrying out logic verification and confidence fusion through a tenant exclusive rule and a rule engine to generate a final auditing state; and updating the data state library, and triggering an alarm or notification mechanism. Therefore, the problems that in the prior art, a data isolation mechanism is weak, auditing rule configuration is rigid, and auditing capacity is single are solved.

A line replaceable unit (LRU) includes a data communication interface configured to receive identification (ID) data from an aircraft communication bus, an aircraft systems interface configured to receive aircraft systems data corresponding to an operation of at least one aircraft system, and an LRU data interface configured to receive LRU data corresponding to the LRU. The LRU further includes a processor and a memory unit. The processor processes the aircraft systems data and the LRU data and determine at least one data category. The memory unit includes a plurality of data partitions. The processor is configured to generate at least one configuration data package by combining the ID data with one or both of the aircraft systems data and the LRU data, and to store the configuration data package in a given data partition based on the data category.

The invention discloses a business request data query method and device, equipment, a medium and a product. The method comprises the following steps: performing request analysis on at least one obtained target service request to obtain a service request condition in the target service request; if the service request condition exists in the pre-constructed data partition mapping table, acquiring a cold and hot data identifier corresponding to the service request condition from the data partition mapping table; and according to the cold and hot data identifier, determining a target data partition and querying the target service request data from the target data partition. According to the scheme of the embodiment, the mapping relation between the service request condition and the cold and hot data identifiers is obtained according to the pre-constructed data partition mapping table; and the target data partition is determined according to the cold and hot data identifier, and the target service request data is queried, so that the data access performance in the distributed database is improved, and the storage efficiency and the storage resource utilization rate of the distributed database are improved.

The invention belongs to the technical field of data storage and query. According to the distributed database system based on the low-difference-degree sequence and the data partitioning method, a coordination node executes data representative partitioning based on the low-difference-degree sequence on written data to obtain a plurality of data fragments, and the data fragments are distributed to all data nodes for storage and construction of a local index; broadcasting a query task to all the data nodes according to a received query request, collecting local retrieval results returned by all the data nodes, performing global aggregation and sorting, and returning the results to a user or an application program; and the data node receives and persistently stores the data fragments distributed by the coordination node, constructs a local vector index for the persistently stored data fragments, performs local retrieval in response to a query task of the coordination node, and sends a retrieval result to the coordination node. According to the method, high-performance distributed database retrieval is realized.

The invention discloses a data partitioning method and device, electronic equipment and a storage medium, and relates to the technical field of computers, and the method comprises the following steps: dividing a sliding window of to-be-processed data into a plurality of sub-regions, comparing adjacent byte pairs in the sub-regions in parallel by utilizing a vector instruction to generate a local maximum set, vector maximum comparison is carried out on a local maximum set through a tree structure so as to reduce the scale of the set layer by layer, finally, a global extreme value is determined to serve as a data block boundary, efficient data block division based on vector parallel calculation is achieved, byte-by-byte or block-by-block serial calculation logic is not adopted, and therefore efficient data block division is achieved. The technical problem that in the prior art, due to the fact that serial computing logic is adopted, computing efficiency is limited, performance bottleneck occurs, and then the processing speed and the resource utilization rate of applications such as data deduplication, compression and encryption are affected can be solved. The technical effects of improving the calculation efficiency, reducing storage fragmentation, relieving the performance bottleneck of a high-concurrency scene and improving the data processing speed and the resource utilization rate are achieved.

The invention discloses a data aggregation processing method and device, computer equipment and a storage medium, belongs to the technical field of big data, and is applied to data aggregation in the financial field. The method comprises the following steps: firstly, grouping dimension fields by utilizing a mapping rule and a hierarchical clustering algorithm; and secondly, dynamic grouping mapping is established, one-time scanning is performed on the source data table, and each data record is directly attributed to a corresponding grouping structure, so that the data processing efficiency is improved. Meanwhile, a correlation index of the index and the data partition is constructed based on a Hash mapping algorithm. Under the support of an indexed data structure, the aggregation calculation function can efficiently execute operations such as summation and averaging, and rapid aggregation of multi-dimensional data is realized. Generally speaking, according to the method, the correctness of an aggregation result is ensured, meanwhile, the calculation and storage pressure of the system is remarkably reduced, the timeliness and expandability of data analysis are improved, and the method is particularly suitable for real-time or quasi-real-time multi-dimensional analysis requirements in a large-scale business scene.

The invention relates to a data recovery method and device, equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that in response to the fact that a terminal is in a first operating system boot starting state, target data in a first data partition is determined; generating a block device mapping graph of the target data, and storing the block device mapping graph to the target data partition; in response to the condition that the terminal is in a factory setting restoring state, acquiring a block device mapping graph from the target data partition by using a second operating system; reading the target data from the block device where the first data partition is located into a memory according to the block device mapping graph; and recovering the target data to the first data partition by using the second operating system. According to the method, the block device mapping graph is stored in the target data partition, so that the second operating system can reserve the target data based on the block device mapping graph, important data are reserved and recovered when factory settings of the terminal are recovered, and it is guaranteed that the data are available and not lost.

The invention discloses a data division method, a large model reasoning method, devices, equipment and a medium, which are applied to the technical field of data processing, and the data division method comprises the following steps: based on historical access information corresponding to each key value cache, utilizing a prediction model to carry out prediction to obtain the probability that the key value cache associated with each key value cache is accessed in the future; dividing the key value caches based on the access frequency parameter, the time interval and the future access probability of each key value cache, and determining a hot data set and a cold data set, so as to store the key value caches in the hot data set to the video memory equipment with the fastest speed. According to the method, the access frequency is considered from the global aspect, the time interval is considered from the time local aspect, the future access probability is considered from the space local aspect, and the data is divided from different aspects, so that the key value can be obtained from the accurate hot data set in time during reasoning, and the accuracy of the data is improved. Therefore, the reasoning speed of the large model is improved.

This disclosure is directed to systems and methods of data partitioning in image encoding. A video encoder may receive a single stream of video data that includes multiple layers. The video encoder may encode the one or more layers utilizing multiple counters for each component within the layers. The multiple counters may correspond to the header bits, luma bits, and chroma bits within each slice layer. The encoded layers may then be assembled into a single slice before it is sent to a decoder for decoding of image frame.

This invention provides a data storage method and system for tuberculosis detection, relating to the field of tuberculosis detection technology. The data storage method for tuberculosis detection includes the following steps: S1. Collecting whole-genome sequencing data of Mycobacterium tuberculosis, host serum IgG titer, and drug sensitivity test results; S2. Calculating genetic distance D based on a reverse evolution model to generate four-dimensional spatiotemporal coordinates (t, x, y); S3. Performing data partitioning and storage based on the drug target barrier value β; S4. Generating dynamic metadata using a host-pathogen dynamics model and compressing and storing it. This invention implements a dynamic storage entropy adjustment algorithm at the hardware and software collaborative level, continuously optimizing the matching efficiency of data compression and physical storage. This results in an intelligent data hub that can perceive the evolutionary pulse of pathogens and autonomously optimize resources, providing support for clinical tuberculosis prevention and control decisions with temporal depth, spatial correlation, and risk evolution.

The invention discloses a file storage processing method and system based on artificial intelligence, and the method comprises the steps: obtaining access description data of a file entity in a file storage data set, and the access description data comprises an access activeness parameter and access result data; mapping the data partition to a storage unit of a corresponding activeness interval based on the activeness parameter; controlling each storage unit to construct access sequence structure information containing sequence and non-sequence arrangement description; generating active and inactive file corresponding relation result data based on the information; evaluating the access change of the target scene file in combination with the multi-interval result to obtain an access change measurement result; therefore, the storage structure is adjusted, the storage resource configuration is matched with the scene file access change characteristics, and the storage resource utilization rate and the access performance are improved.

Embodiments of this specification disclose graph data partition computer-implemented methods, non-transitory, computer-readable media, and computer-implemented systems. A computer-implemented method includes partitioning vertices in graph data into a plurality of dataset. Edges in the graph data are partitioned into datasets that include target vertices of the edges, where the datasets are used by nodes in a distributed cluster to perform graph computation, and where computational loads of the plurality of datasets are similar. Implementations of this specification can achieve load balancing between nodes in the distributed cluster and can reduce communication overhead.

Systems, methods, and devices for data ingestion, database management, and data security. A method includes aggregating data from a plurality of different data sources, wherein the data comprises equipment data and personnel data pertaining to a project. The method includes generating a longitudinal data bucket comprising information associated with the project for tracking the equipment data and the personnel data over time and storing the longitudinal data bucket on a database. The method includes assigning permissions to different types of data stored in the longitudinal data bucket such that a plurality of users of the database have personalized read and / or write permissions to the equipment data and the personnel data.

This invention belongs to the field of soft measurement modeling technology and discloses a soft measurement modeling method based on mode-related spatiotemporal diffusion generation, which includes the following steps: (1) acquiring data of multi-mode dynamic processes; (2) data partitioning and preprocessing operations; (3) establishing a mode-related spatiotemporal diffusion model and generating virtual samples; (4) predicting pressure variables of three-phase flow processes and evaluating model performance. This invention proposes a soft measurement modeling method based on mode-related spatiotemporal diffusion generation. By using the DSTN-Net noise prediction network to capture the dependence of dynamic data in the time and space dimensions, the mode learner learns the multi-mode distribution characteristics and distinguishes the data distribution of different modes, which can generate virtual samples with high similarity to the original samples, thereby improving the prediction performance of the soft measurement model in the case of small samples.