183 results about "File size" patented technology

The invention discloses a large file concurrent uploading method and system based on a dynamic fragmentation strategy and resource optimization, and the method comprises the steps: a client generates a file fingerprint according to the file size of a large file and metadata, and dynamically calculates the fragmentation size of the large file; the client carries out fragment cutting on the large file according to the calculated fragment size, establishes a concurrent uploading queue and uploads a plurality of fragments to a server side in parallel; the server selects an optimal storage node to store each fragment based on a consistent hash ring and a multi-dimensional weight model; the client side and the server side track a fragment uploading state through a fragment serial number and a cyclic redundancy check (CRC) value, and after all fragments are uploaded, integrity check and logic recombination are carried out; and when network interruption or uploading failure is detected, the client side and the server side execute an intelligent retry strategy or a splitting rescheduling strategy of breakpoint resume and failure fragmentation.

The embodiment of the invention discloses a file fragment uploading method and device, electronic equipment, a computer program and a storage medium. According to the embodiment of the invention, a to-be-uploaded file and file information are obtained, wherein the file information at least comprises a file size; segmenting a to-be-uploaded file according to the initial network speed of the current network to obtain a plurality of initial file fragments; uploading the initial file fragments to the server in sequence, and recording uploading time consumption and uploading rate corresponding to the initial file fragments; adjusting the size of the next initial file fragment according to the uploading time consumption and the uploading rate of the uploaded initial file fragment to obtain a target file fragment; and uploading the target file fragments to the server. Therefore, in the file uploading process, the real-time network state is determined according to the uploading condition of the uploaded fragment by adopting a mode of'dynamic fragment uploading + network self-adaption ', and the size of the fragment is dynamically adjusted according to the real-time network state, so that the file uploading success rate and the user experience are improved, and the method is flexibly applied to various large file uploading scenes.

The invention discloses a distributed document analysis method and device, equipment and a medium. The method comprises the steps that a target to-be-analyzed document is acquired; extracting content feature data of the target to-be-analyzed document; the content feature data is used for reflecting the content complexity of the target to-be-analyzed document; according to the content feature data, determining estimated time consumption for analyzing the target to-be-analyzed document; based on load balancing, according to the estimated time consumption, distributing a target parsing task of the target to-be-parsed document to a target computing power node; and the target computing power node executes the target analysis task to obtain an analysis result. Therefore, the analysis duration is pre-estimated based on the content feature data of the document instead of determining the pre-estimated time consumption singly depending on the size of the document, the method is adaptive to the time consumption fluctuation characteristics of different documents, the precision of the pre-estimated time consumption is improved, calculation power stage distribution is performed based on load balance according to the pre-estimated time consumption, and the calculation efficiency is improved. The resource utilization rate and the load balancing effect of the system are improved.

The invention relates to a Json file parallel analysis and data writing method and device based on DataX. The method comprises the following steps: acquiring a source end Json file set, and dividing the source end Json file set into small file subsets and large file subsets according to file sizes; the small file aggregation path is configured with a json reader of the DataX, and parallel reading and analysis are carried out; splitting a root array of the large file into a plurality of independent small Json files through a Python script, aggregating paths, and reading and analyzing by using a DataX parallel channel; and finally executing the DataX Job to write the data into a target end. Through a classification processing strategy, the large-scale Json data analysis writing efficiency is remarkably improved, the large file memory occupation risk is reduced, the file scale is automatically adapted, a mature DataX frame and a Python script are reused, high performance and low maintenance cost are both considered, multi-type target end storage is supported, and the problems that a traditional tool is weak in parallel, high in resource consumption, poor in flexibility and the like are effectively solved.

The invention discloses a file access method, device and equipment and a computer readable storage medium, and belongs to the technical field of storage. The method is applied to a first node in a file system and comprises the steps that a first write request for a file is received, the first write request comprises first file data and IO information, and the IO information represents an address range for writing the first file data; in response to the first write request, metadata of the file is obtained, and the metadata comprises an extension attribute and a file size; according to the IO information and the file size, under the condition that it is determined that the written file size does not exceed the reference threshold value, the first file data is written into the extension attribute, and the written file size is the file size after the first file data is written. The file data is embedded in the metadata of the file, and the file data can be accessed while the metadata of the small file is accessed, so that the step of accessing the file data is reduced, the response time delay of the file system to the access request is reduced, and the purpose of improving the access performance of the system is achieved.

The invention relates to a DataX-based FTP (File Transfer Protocol) unstructured file transmission breakpoint resume implementation method and device. The method comprises the steps that a progress tracking module is integrated in a DataX Writer, and a transmission progress metadata file (TPMF) containing information such as the number of uploaded bytes and the state is created for each file and dynamically updated; deploying an independent process monitor to monitor task interruption, and triggering progress persistence to TPMF; and a user selects a file through the independent FTP continuous transmission scheduler, reads the TPMF and the size of a target file at a server side to judge a breakpoint, calls an FTP REST command to perform continuous transmission from the breakpoint, and updates the state after the continuous transmission is completed. According to the method, accurate progress recording, abnormal interruption active storage and user controllable continuous transmission are realized, redundant retransmission is avoided, the transmission efficiency and the resource utilization rate are improved, the data synchronization robustness is enhanced, and the operation and maintenance complexity is reduced. The method is small in DataX core invasiveness, high in universality and suitable for high-reliability synchronization scenes of large-scale unstructured files.

The invention relates to a three-dimensional model fingerprint generation method and system, a terminal and a storage medium. The method comprises the following steps: identifying a file type, analyzing a file structure for the file type containing structured entity information, and extracting triangular mesh entity data; executing a block sampling strategy based on the file size for the file type which does not contain the structured entity information; counting the total number of triangular facets for the triangular mesh entity data, and selecting an adaptive hierarchical sampling strategy according to the total number of the triangular facets; triangular surface area sampling is carried out on a triangular mesh entity, a sampling block-jumping block alternating traversal mode is adopted, and compared with an existing mode, the performance is improved by 69-83 times; besides, a two-level Hash structure is adopted, the first level is to calculate the MD5 value of the area accumulated sum for each sampling block, the second level is to calculate the overall MD5 value for the connection sequence of all block level MD5 values, and the two-level structure enhances the uniqueness and collision resistance of fingerprints.

The application relates to the technical field of electronic equipment, in particular to a method for flashing an operating system, which comprises the following steps: generating restriction information containing a customer identification ID and an image validity period, encrypting the restriction information as a file header through the customer identification ID, forming a double-encrypted file by encrypting the main body of the image with the restriction information; binding a flashing tool with the customer identification ID and hardware information of a flashing device, decrypting the file header and checking the consistency of the customer identification ID and the image validity period in the restriction information after checking the legality of the binding; decrypting the main body of the image and burning it to a target device after the checking, and clearing temporary data after the burning is completed. The double-encryption mechanism prevents the image from being leaked and tampered with, the hardware binding realizes the authorization checking in an offline scene, the multi-dimensional checking such as the validity period and the file size precisely controls the use range of the image, and the safety and controllability of the flashing process are effectively improved.

The application provides a lightweight log generation method, system and electronic equipment, and relates to the technical field of computer software.The method comprises the following steps: creating a running log management task and an operation log management task with lower priority than an application task, and establishing a message queue; when a log event is generated by the application task, a formatted log message containing a timestamp, a log level, a module identifier and content is generated and sent to the message queue; the log management task asynchronously acquires the log message, and distributes the log message to a running log folder or an operation log folder according to the log level and type; and the size of the log file is monitored, and when the capacity threshold is reached, a cover writing mechanism is enabled to recycle the file storage area.The application realizes asynchronous decoupling of log recording and the application task, supports hierarchical classified storage and cyclic coverage, significantly improves the maintainability and debugging efficiency of the embedded system while ensuring real-time performance, and meets the lightweight log demand in a resource-limited environment.

The invention relates to the technical field of audio quality evaluation, in particular to a multi-dimension-based audio quality evaluation method, which uniformly quantifies file size, tone quality loss, coding and decoding complexity and multi-scene applicability into a single comprehensive score, and obtains an audio quality evaluation result through an adjustable sensitivity factor, dynamic slope amplification and a geometric mean short plate sensitivity mechanism. And meanwhile, the smooth constant epsilon ensures the numerical stability of logarithm operation, and quick, objective and automatic audio quality batch evaluation is realized. Therefore, the problem that an existing audio quality evaluation system is single in dimension is solved.

The invention provides a file identifier display method, electronic equipment and a storage medium, and the method comprises the steps: determining a target folder from a plurality of information folders according to a to-be-queried file path included in a file identifier display request; determining a primary folder from a plurality of sub-folders in the target folder; establishing a primary folder identifier corresponding to each primary folder; performing congeneric clustering on the plurality of primary folders to obtain a plurality of primary folder class groups; establishing a folder identifier of a folder corresponding to each secondary folder identifier; and arranging the folder identifiers corresponding to the sub-folders in the target folder to form a file identifier hierarchy graph and displaying the file identifier hierarchy graph. According to the method, the display interface of the file identifier display system displays the primary folder identifiers as many as possible under the condition that the size of the canvas is fixed, and the sizes of the files included in the corresponding information folders can be intuitively reflected by checking the radiuses of the primary folder identifiers.

The application relates to the technical field of data compression storage, and provides a high-throughput genome sequencing quality score data parallel compression method.The method comprises the following steps: segmenting an original gene sequencing file; performing random sampling and k-mer analysis on the sampling data to obtain statistical characteristic information and establish a parallel sequence partition model for binary classification; splicing two partition files obtained through binary classification according to splicing parameters; predicting the to-be-compressed file through a multivariate linear regression analysis prediction method to obtain compression rate gain and establish a parallel four-level run prediction mapping model for data redundancy elimination; and performing context modeling on two redundancy-eliminated subfiles through a multi-core processor cluster, and performing cascade compression in combination with arithmetic coding to obtain a final compressed file.Under the premise of significantly reducing quality score data compression time and peak memory overhead, the application also improves the quality score data compression rate, reduces the size of to-be-compressed storage files, and saves the construction cost of basic storage facilities.

The invention discloses a cache optimization method and device based on distributed storage, and relates to the technical field of distributed storage systems. The method comprises the following steps: dynamically calculating a weight factor of a cache file, attenuating the weight factor based on the access frequency and timeliness of the file, and compensating and adjusting according to the size of the file; the maximum file threshold value set by the system is dynamically adjusted according to the size distribution of the files in the cache, and the maximum file threshold value is determined based on a preset quantile range so as to eliminate interference of extremely large files on weight calculation; when the weight factor of the cache file is lower than a preset threshold value, the file is removed from the cache, and the cache space is released; when the cache capacity is insufficient, the file with the minimum weight is selected for replacement based on the weight factor of the current cache file, and the high-value file is preferentially reserved in the cache. According to the method, the cache hit rate of small files in the distributed storage system is effectively increased, cache space waste and metadata operation overhead are reduced, and system performance and resource utilization rate are remarkably improved.

The application provides a firmware fast loading method and circuit of an optical-electric DSP chip and an optical module. The method is applied to a DSP unit which is installed in the optical module and is used for executing a starting program. The DSP unit is electrically connected with an MCU unit of the optical module and obtains loading information of the MCU unit. The method comprises the following steps: obtaining first information and obtaining second information according to the first information. The file size of the first information is smaller than that of the second information, and the first information contains communication parameter information used for obtaining the second information. The second information contains target firmware of the DSP unit. The application can improve the loading speed of the firmware of the DSP chip, reduce the development work of the control program of the MCU and has good compatibility on the basis of saving hardware cost.

The invention belongs to the technical field of enterprise information security, and discloses a technical method for a company device to automatically back up and clear sensitive information files according to a list, which comprises the following specific steps of: 1, dynamically monitoring the list, deploying a lightweight monitoring execution engine by a terminal, and monitoring an interface through a system-level file; the change of a list file of a specified directory of the desktop is detected in real time, the monitoring parameter is set as polling once per second, and the analysis engine is triggered after the size of the file is detected to be unchanged for 2 seconds; an automatic closed loop of millisecond-level response is constructed through an Excel list driving engine, desktop list changes are captured in real time through an operating system-level monitoring interface, thousand-row-level Excel analysis (precisely extracting A-column paths / B-column instructions) is completed within 60 seconds, follow-up operation is automatically triggered, and therefore the file processing time of a single device is shortened to be within 5 minutes, and the file processing efficiency is improved. The misoperation rate is reduced to 0.1% or below, and response lag and errors caused by human factors are thoroughly eliminated.

The invention discloses a multi-modal medical image compression and decoding method and system based on an implicit neural representation fusion hyperfunction. The multi-modal medical image compression and decoding method based on the implicit neural representation fusion hyperfunction comprises four steps. According to the method, the integrity of image structures and details can be kept while the high compression ratio is realized, and the quality requirements of medical images in clinical diagnosis and scientific research analysis can be met. Moreover, the method has the advantages of high compression ratio, high cross-modal adaptability and task performance retention capability, and is suitable for data compression and remote application scenes of multi-modal medical images such as CT, MRI, PET and pathological images. According to the method, the file size can be reduced by more than 90% after compression, and Dice coefficient expression higher than that of an existing method is obtained in downstream tasks such as image segmentation.

The application discloses a multimedia file storage method, device, equipment and medium, wherein the multimedia file storage method comprises the following steps: obtaining a live stream of a multimedia file; converting the live stream into a plurality of block files in a target format, wherein the size of the block files changes with the length of a cache queue and / or the number of writable disks; sequentially writing the plurality of block files into the cache queue, and scheduling a plurality of threads in a thread pool to write the block files in the cache queue into the writable disks in parallel, the optimal block file size can be calculated according to the length of the cache queue and the number of the writable disks, then the block files are read from the cache queue and written into the writable disks in parallel through thread scheduling, so that the threads reach the optimal scheduling frequency and the disks reach the optimal writing speed, and the efficiency of writing the live stream into a disk is effectively improved.

The application discloses a data storage method, system and device, a storage medium and a program product, and belongs to the field of computer data processing. The method comprises the following steps: receiving distributed storage data sent by an execution node, wherein the distributed storage data is data stored by the execution node in a distributed storage manner; transferring the distributed storage data to a local disk for storage, wherein the local disk is a local storage area corresponding to the current device; in response to the local disk meeting data uploading conditions, uploading local storage data in the local disk to a cloud storage system according to a preset uploading strategy, wherein the preset uploading strategy is a strategy determined based on the average file size in the local disk; and in response to the uploading of the local storage data being completed, cleaning the local storage data from the local disk. The method achieves the effect of cooperation between local storage and the cloud storage system, and solves the problem that the large data workload cannot be effectively deployed in an offline hybrid cluster due to the fact that storage and calculation cannot be separated.

The application discloses a distributed document analysis method, device, equipment and medium, the method comprises the following steps: obtaining a target to be analyzed document; extracting the content feature data of the target to be analyzed document; the content feature data is used to reflect the content complexity of the target to be analyzed document; according to the content feature data, the estimated time consumption of analyzing the target to be analyzed document is determined; based on load balancing, according to the estimated time consumption, the target analysis task of the target to be analyzed document is distributed to the target computing power node; so that the target computing power node executes the target analysis task to obtain the analysis result. Therefore, the content feature data of the document is used to estimate the analysis time, rather than relying on the file size to determine the estimated time consumption, which is adaptive to the time consumption fluctuation characteristics of different documents, improves the estimated time consumption accuracy, and based on load balancing, the estimated time consumption is used for computing power stage distribution, which improves the resource utilization and the load balancing effect of the system.

A method and device estimate available transmission bandwidth of a network connection used to deliver a data file within a predetermined transmission time. A file size is obtained and a minimum average bitrate required to meet the time is computed as size divided by time. The transmission window is partitioned into alternating first and second time intervals. During each first interval, portions of the file are sent at a bitrate greater than the minimum average. During each second interval, remaining portions are sent at a lower bitrate selected such that the overall average bitrate across the window is at least the minimum average and below a configured bitrate threshold. Feedback (e.g., acknowledgments, loss, delay, or throughput) collected during the higher-rate first intervals is used to estimate the available bandwidth of the connection. The estimate may be used to adjust subsequent interval rates, file scheduling, or admission decisions.

The invention provides a data processing method and system, a storage medium and a program product. The method comprises the steps of obtaining a to-be-uploaded target file; for a first file of which the file size is greater than a file size threshold value in the target file, uploading the first file to a search resource server by adopting a fragment uploading mode; and for the second file of which the file size is smaller than or equal to the file size threshold value in the target file, directly uploading the second file to the search resource server, and selecting a better uploading path for the files of different sizes through simple threshold value judgment. The small file avoids the overhead of multiple rounds of handshake and multiple requests in fragment uploading, and the small file is quickly uploaded directly through one request, so that the delay caused by network round-trip time is greatly reduced.

The invention relates to a bandwidth control method and device in a file preheating process and electronic equipment. The method comprises the following steps: acquiring the size of a current pre-issued file in a queue to be preheated, the size of an issued file in a current bandwidth control period and a size threshold value of the issued file in the current bandwidth control period; under the condition that the sum of the size of the current pre-issued file and the size of the issued file is lower than or equal to an issued file size threshold value, determining the sizes of target issued files corresponding to the plurality of content distribution nodes according to the size of the current pre-issued file; the sum of the sizes of the multiple target issuing files is the size of the current pre-issuing file; and respectively sending notification information of the corresponding target issuing file size to the plurality of content distribution nodes, so that the plurality of content distribution nodes control the bandwidth for obtaining the target file content in the to-be-preheated file from the source station corresponding to the to-be-preheated file according to the target issuing file size. According to the method, the preheating bandwidth can be well controlled, and a certain preheating speed can be guaranteed.

This invention relates to a distributed video file processing method and system based on RAID disks via FTP. The method includes: calculating the size, content hash value, and block content hash value of the video file to be uploaded via an FTP client and uploading it to an FTP scheduling module; the FTP scheduling module searches the FTP server for a file with the same size and hash value as the video file to be uploaded; if a file exists, the FTP file address of the same file is returned to the client; if no file exists, an optimal FTP server address is selected and returned to the client, and the client uploads the video file to the optimal FTP address. This invention utilizes the fast read / write characteristics of RAID 0 and also adds an idle backup and recovery file module to ensure data security. If a file is corrupted, a backup file is used for download, and the file can be recovered in the background.

The application relates to the technical field of computer application, and discloses a mass file data stream storage method, a storage system and a big data processing system. A rotation state conversion model is arranged between a data stream and a storage node, and storage pressure is dispersed through multi-node caching. The storage method comprises the following steps: presetting a file size demarcation point; inputting mass file data; judging whether the size of the input file is smaller than or equal to the demarcation point; storing file data smaller than or equal to the demarcation point in a selected storage node through the rotation state conversion model; generating index information of all small files, and merging the index information into an index file; organizing data in the form of a B+ tree for each index file to obtain an index data bucket; storing a plurality of index data buckets in the form of data blocks in the storage node; and saving all index data buckets in sequence as an index data bucket queue. The application can effectively solve the problems of traditional distributed storage systems, such as write performance decline and low efficiency when facing mass files.

This application provides a resource scheduling method, apparatus, electronic device, and storage medium for a hybrid storage system. The method includes: acquiring a set of files to be scheduled in a hybrid storage system, wherein the hybrid storage system includes hard disk drives (HDDs) and solid-state drives (SSDs); for each file to be scheduled in the set, determining the random access frequency, access popularity, file size, and read operation ratio of the file to be scheduled in the current period, wherein the read operation ratio is used to characterize the proportion of read operations to the total access volume; determining the file score corresponding to the file to be scheduled based on the random access frequency, access popularity, file size, and read operation ratio; and storing all files to be scheduled to the corresponding storage location according to the file score of each file to be scheduled. This achieves both reduced storage device costs and improved data access speed of the storage device.

The invention relates to a cloud database computing cache system and an implementation method, and the system comprises a decentralized cache architecture module which is used for eliminating a centralized bottleneck, supporting ten-thousand-level concurrency and adapting to a big and small file mixed scene; the memory object sharing module is used for reducing intermediate result processing delay through intermediate result full-life-cycle zero-copy circulation and dynamic storage decision based on a memory pool and metadata service of the memory object sharing module; the self-adaptive asynchronous write-in module is used for connecting remote storage, multiplexing memory object sharing module resources and combining mechanisms such as delay confirmation, so that the write response speed is increased, remote requests are reduced, data integrity is guaranteed, and meanwhile, persistent support is provided for intermediate results. Through cooperative operation of the three modules, the cloud database computing cache system and the implementation method can effectively adapt to the requirements of high concurrency, large and small file mixing and elastic capacity expansion and contraction of a cloud native OLAP scene, and the integrity of data transmission and the stability of system operation are guaranteed.

A smart summary system, device, and storage medium for program backup results of large-scale heterogeneous device clusters includes: a table-based human-computer interaction interface provided on a host computer for inputting and managing backup information of large-scale heterogeneous devices; after backup, the system performs a first-stage difference screening on each program file, comparing the current file size with the size of the file with the same name from the last backup; if the sizes are inconsistent or the file is newly added, it is marked as a difference file, and detailed records are generated for the marked difference files; the system performs statistical analysis on the complete difference log, generates a smart summary report, and actively pushes the smart summary report to the administrator through multiple communication interfaces. Existing common program backup methods suffer from technical problems such as extensive management, low efficiency, unintuitive results, and delayed decision-making.

The invention relates to a file storage method and device, electronic equipment and a storage medium. The file storage method comprises the following steps: receiving a file uploading request of a user; determining a target data block of which the storage granularity is matched with the size of a to-be-uploaded file according to the size of the to-be-uploaded file in the file uploading request; and when it is determined that the remaining storage units exist in the target data block, writing the to-be-uploaded file into the target data block, and storing metadata of the to-be-uploaded file and the first storage position information into a preset database. The data blocks can be classified according to the storage granularity, the target data blocks of which the storage granularity is matched with the size of the to-be-uploaded file are determined according to the size of the to-be-uploaded file, and the files belonging to the same storage granularity are stored in the same data block, so that the storage efficiency is improved, the management of metadata is facilitated, and the user experience is improved. Therefore, the storage performance is improved under the condition that hardware does not need to be improved.

The invention relates to the technical field of data processing, and particularly provides a front-end processing method, system and device for a large file and a storage medium, and the method comprises the steps: determining the size and the standard format of a standard file; segmenting the large file into a plurality of blocks according to the size of the standard file, and converting each block into a standard format; generating a configuration table based on large file segmentation and format conversion, wherein the configuration table comprises processing metadata, large file information, block information and a check value; and sending the blocks and the configuration table to a back-end server, so that the back-end server verifies and integrates the blocks according to the configuration table. According to the method, the bottlenecks of fault tolerance, adaptability and safety in large file transmission are remarkably solved.

The invention discloses a wireless hard disk based on double-module transmission and a wireless hard disk data interaction method, and relates to a hard disk data transmission technology. The magnetic sensor monitors the environmental magnetic field intensity, and when the environmental magnetic field intensity exceeds a set threshold value, a wake-up signal is sent to the main controller; the UWB positioning module calculates the current position coordinates of the hard disk, the motion sensor detects a real-time deviation value and feeds back the real-time deviation value to the compensation controller, and the charging power is adjusted according to the real-time deviation value; judging file sizes, and switching to corresponding data transmission modes based on different file sizes; and if the magnetic sensor does not detect the effective magnetic field for a continuous period of time and the distance measured by the UWB positioning module exceeds a set threshold, the hard disk is determined to leave. The problems that an existing wireless hard disk is low in charging efficiency and unstable in data transmission during position deviation are solved.