817 results about "Small files" patented technology

Grouping and navigating media files within a playlist on a computer-readable medium. Invention software selects media files according to a grouping criterion to define a group from the media files. The invention software generates a group header associated with the group. The group header includes a value identifying the defined group and a reference to another group to enable navigation. The invention software stores the selected media files, the playlist, the defined groups, and the associated group headers on a computer-readable medium. The invention software also includes playback software for navigating and selecting media files via groups within a playlist. In one embodiment, the invention is operable as part of a compressed media format having a set of small files containing metadata, menus, and playlists in a compiled binary formal designed for playback on feature-rich personal computer media players as well as low cost media players.

A flat file data organization technique is used for storing and retrieving geospatially organized data. The invention reduces transfer time by transferring a few large files in lieu of a large number of small files. It also moves the process of locating a given data file away from the file system to a proprietary code base. Additionally, the invention simplifies database management by having quadtree packets generated on demand.

A local user issues recording instructions via the Internet to a remote computing device. The remote recording device includes a computer daughter board that records broadcast programs available to it according to the received recording instructions and encodes the recorded program into a specified media format. The daughter board further adds a first field identifying the user and a second field listing some of the received recording instructions on the recording. If the recording is video, then the fields are placed on visible sections of the recording. The file holding the encoded broadcast program may be subdivided into multiple smaller files before being moved into another computer specified by the user and accessible via the Internet. The remote computing device may receive recording instruction via a pulse telephone and may be incorporated into existing video cassette recorders.

The invention discloses a small file optimization storage method based on a HDFS, wherein the efficiency for the HDFS to read small files is improved, and the overall performance of the system is improved. The method includes the steps that first, the small files are combined and undergo storage preprocessing, wherein the storage preprocessing on the small files is achieved through filtering of the filers, combination of the small files, generation of metadata and generation of object IDs; second, after the files are stored into the HDFS in a combined mode, the mapping relations between the small files and combined files in the HDFS are stored into the metadata of the small files in a mode of file metadata, a directory structure of the files is stored in a file name, and the metadata are stored in a mode of distributed clusters on the basis of a Chord protocol; third, the directory structure of the files is optimized, and generated key values of the metadata are decomposed into Directory IDs and Small File IDs. The Directory IDs serve as key values for the metadata to skip into nodes in a metadata cluster, and therefore the files under the same directory are stored into the same node. The Small File IDs are generated in metadata nodes, and therefore each of the metadata corresponds to unique ID identification in the whole system.

The invention relates to a computer-based method and system to facilitate quality control for manufactured assemblies based on computer aided design (CAD) files. The instant invention mitigates the problems encountered with large CAD files by decomposing each of such files into a multi-file format. A large CAD file is thereby broken-down into smaller files that organize the information contained in the larger file. The organization is performed in such a way that the information pertinent to the measurement process is segregated into a different smaller file than the information not needed to calculate measurements. Thus, the computer running the computation accesses a smaller file containing a higher percentage of required information. Additionally, assisting the user in coordinating a match-up between the physical surfaces being measured and the proper associated CAD model version of that surface further increases the handling speed. In particular, the instant invention uses selection regions for each surface. A selection region consists of the representation of a three-dimensional rectangle region just large enough to enclose each individual surface. A selection region indexes each surface. Thus, when a measurement point is taken, a list of surface regions is automatically scanned in order to determine which region contains that point. The corresponding distance of that measurement point to the surface is then calculated.

The invention discloses a Hadoop-based mass classifiable small file association storage method, which mainly solves the problem of low access efficiency of classifiable small files. The method comprises a NameNode global index management technology and a file clustering technology. Independent small files belonging to a certain class are subjected to file clustering and global index management, so that the utilization rate of an internal memory is greatly improved and the maximum file number supported by unit internal memory is improved. The method comprises the following steps of: (1) clustering small files belonging to a certain class into a file called a logic unit; and (2) establishing a global index for each small file stored in a NameNode internal memory. The file clustering technology is used for improving the storage efficiency of the classifiable small files, and the NameNode global index management technology is used for managing the clustered small files. By the technologies, the storage efficiency of the mass classifiable small files is improved. The invention is suitable for storing and managing the classifiable small files under general scenes.

The invention discloses a mass non-independent small file associated storage method based on Hadoop. The method is mainly used for solving the problem of low mass non-independent small file access and reading efficiency and aims at a plurality of small files, namely non-independent small files obtained by cutting a big file. The method is characterized by comprising the following steps: (1) merging all the small files of the big file into one file which is named merged file; (2) establishing a local index for each merged file, and storing a local index file and a file entity onto a Data Node of a Hadoop system while updating; and (3) when the non-independent small files are read, improving the file reading efficiency by adopting metadata cache, local index file pre-fetching and associated file pre-fetching. By utilizing the method provided by the invention, the efficiency of the existing Hadoop system for storing and reading small files is improved. The method is suitable for the storage and management of the mass non-independent small files in universal scenes.

User Relayed Broadcasting (URB) software creates a media file segmentation and distribution system for affordable broadband live media broadcasting over the Internet to vast audiences. It solves the bandwidth problem for live broadcast servers, eliminating the need to choose between quantity and quality when broadcasting live media over the Internet. URB receives a data stream from a conventionally-encoded media source, segments it into small files, uploads the files to users who re-upload them repeatedly in a chain-letter style multiplier network, and then plays the files back continuously through a conventional media player half a minute later. In effect it only simulates live broadcasting—it isn't live and it isn't broadcasting. It is file-sharing, or rather file-distributing, of media files using a very brief create-distribute-redistribute-play cycle and a time-synchronization protocol. Put simply, URB combines features of (1) live web-based media, such as Shoutcast or Icecast; (2) a file-sharing system similar to Gnutella or Napster; and (3) a cyberspace time-synchronization protocol.

The invention provides an optimized method for accessing lots of small files for Hadoop, and aims at providing a small file merging, indexing and querying method applied to the Hadoop. The method mainly solves the problem of low access efficiency of small files in the Hadoop. The invention provides a three-layer Hadoop small file access processing architecture; and the three layers are respectively a user interface layer, a service logic layer and a data storage layer. According to the method, a merging mapping technology of the small files at a preprocessor side and a fast indexing technology of lots of small files are used.

The invention discloses a method for storing a mass of small files on the basis of a master-slave distributed file system, and aims to solve the problem about storing the mass small files in the master-slave distributed file system. The technical scheme includes that the method includes that a mass small file storage system is deployed and initialized, and then small files are created and read by a SmallFileAPI (small file-application program interface) of a client side according to a command received from a keyboard. When the small files are created by a system, the SmallFileAPI creates data files of the small files according to small file paths obtained from the client side, writes in small file data and simultaneously creates small file indexes at data nodes; when the small files are read by the system, data node information corresponding to a parent directory is acquired according to the small file paths, an index request is transmitted to any of the data nodes, and finally the small file data are read from the data files according to index information. By the aid of the method, the problem about mass of storage metadata of the mass of small files is solved, writing efficiency of the mass small file storage system is improved, and reliability of the system is guaranteed.

The invention relates to the field of an HDFS of a computer, and discloses a method for storing and processing small log type files in a Hadoop distributed file system (HDFS). According to the method, files are combined in a nearby mode according to physical locations, and a Copy-On-Write mechanism is used for optimizing read-write of the small files; specifically, the small log type files are combined in a nearby mode according to a physical path, a client side reads and writes the combined files from a NameNode and Metadata information of indexes of the combined files when reading and writing the small log type files, and then all the small log type file data are read and written from the combined files according to the indexes of the combined files. According to the new processing method of the small log type files, the memory load of the metadata of the small files are transmitted to the client side from the NameNode, and the problem that when the HDFS processes a large number of small files, efficiency is low is effectively solved. The client side caches the metadata of the small files, so that the speed of access to the small files is improved, and a user does not need to send a metadata request to the NameNode when sequentially accessing small files which are adjacent in physical location.

The invention discloses a two-stage single-instance data de-duplication backup method. De-duplication of data at two stages is performed during backup. The method comprises firstly, performing repeated data detection on a file scale, inquiring local logs to judge whether identical files are stored or not, and if the identical files are stored, informing users to complete the backup operation; if identical files are not stored locally, informing backup programs of server ends to inquire databases to judge whether files with identical content exist or not, if the files with identical content are searched, establishing links pointing to the files for clients only, and recording quotes of the files by clients by the server ends; if the files are new, uploading the files and recording information of the files by two ends; further processing the files after the files are uploaded to the server ends by background programs, and splicing small files together to avoid waste of space; storing large files respectively by type, comparing similar files regularly, and performing difference de-duplication at the second stage after grouping.

The invention relates to a data processing method and device 30 in a distributed file storage system. The method includes a client proxy 30 receiving the user data processing request including target file identifier, offset address, file length and other information; the client proxy 30 obtaining redundancy rate information contained in the target file identifier and containing the data strip number N of the distributed file storage system and a verification stripe number M of the distributed file storage system; determining the effective stripe number DSC of the target file according to the offset address and the file length information; determining the actual stripe number N' of the target file based on the effective stripe number DSC and the verification stripe number M; and determining and processing the corresponding stripes according to the actual stripe number N'. The number of the actual stripes generated during striping a target file can be dynamically adjusted according to the size of the file to be processed. In this way, correct target files can always be obtained under any circumstances, and the number of the empty strips in the distributed file storage system 10 can be reduced. Lots of network read-write I/O and disc I/O can be saved under small file conditions, thereby improving the performance of the distributed file storage system 10.

The invention discloses a video processing method and system. The method includes the following steps that: a video file to be processed is obtained at a client end, and the video file to be processed is split into a plurality of segment files, and the plurality of segment files are compressed and are uploaded to a server; at the server end, after being received, all the segment files are de-compressed and stored in a distributed file system; the segment files of the video file to be processed are download and are compressed and decoded, and new segment files are stored in the distributed file system; and the segment files are merged to form a complete video file. According to the method of the invention, at the client end, the video file is split into the small files from a large file, and so that the video file can be segmented to form the segment files, and the segment files are uploaded, and therefore, transmission efficiency can be improved, and network traffic consumption can be decreased; at the server end, the plurality of small files are compressed and decoded under a cluster, so that processing time can be decreased. With the video processing method and system adopted, the video file can be played independently and can be played in a blocked and segmented manner, and therefore, and user experience can be greatly enhanced. The video processing method and system are applicable to the video playing field in complex environments.

The invention relates to a computer HDFS field and discloses an improved method aimed at small files of an HDFS. Partial authority of a Namenode is delegated to a Datanode which caches element data information of part of small files and the Datanode is used for processing the writing and reading request of most small files. Burdens of the Namenode are reduced to the utmost. The new processing method is provided for solving the problem that the efficiency in processing the small files by the HDFS is low. The improved method aimed at the small files of the HDFS can effectively solve the problem that burdens of a single node, namely the Datanode are excessively large and pressure of the small files is distributed to the Datanode. Thus, large file and small file processing efficiency and performance of the integral large data processing colony are quite ideal.

A method of storing files in a data storage device includes dividing a storage medium of the data storage device into identically sized storage units having a predetermined size, and indicating a status of each storage unit and relating chains of storage units storing a same file in a file allocation table on the storage medium; grouping the storage units into a plurality of groups having a predetermined number of adjacent storage units; assigning a file size classification when creating a file to be stored on the storage medium, the file size classification corresponding to an estimated amount of data in the file to be stored on the storage medium; and allocating at least one of the groups of adjacent storage units for storing the file if the file size classification is greater than a predetermined value.

The present invention discloses a Hadoop-based storage optimizing method for small file hierachical indexing. By using the method, a large amount of small files can be uploaded onto an HDFS file system, and the small file indexing efficiency stored on the HDFS file system can be raised. The method mainly comprises a first step that a WebServer monitors file read-write requests and judges sizes of files are smaller than that of blocks set in the HDFS file system; a second step that files with the sizes larger than the size of the blocks are read and written according to normal HDFS files, files with the sizes smaller than the size of the blocks are categorized as small files which are sent to a small file processing server for processing; a third step that the small file processing server is internally provided with a judgement module, and 1-1023KB small files are set as K-level small files while 1M-64M small files are set as M-level small files; and a fourth step of the small file processing server is also provided with a buffer memory, a part of small files can be preset during file separation according to file correlation, and cache updating strategies are designed to raise the file accessing efficiency.

The invention discloses a method and a system for creating files in cache of a distributed file system client. The method includes the steps of firstly, allowing the client to acquire catalogue writing authorization from a metadata server; secondly, allowing the client to create files in local cache; and thirdly, allowing the client to asynchronously submit the created files to the metadata server. The problem that the cost of interaction with a metadata server synchronous network fails to be effectively lowered during file creation for the distributed file system in the prior art under the scenario of application of lots of small files is solved.

The invention discloses a storage system based on a Hadoop distributed computing platform. The storage system comprises an HDFS (Hadoop Distributed File System) general file processing module, a file type judging module, a small file processing module and a timing module, wherein the file type judging module is used for judging whether files uploaded by a user belongs to small files; the timing module carries out timing through setting a timer, accounts the size of medium-small file sequences of the small file processing module when the preset period is reached, and judges whether the size of the small-file sequence is more than a block of an HDFS file system; and the small file processing module is used for storing each small file as a Record in a Sequence File class to form a small file queue. According to the storage system, the quantity of the small files in an HDFS is reduced, and the property of reading the files in the HDFS is effectively improved.

The invention discloses a storage method under a cloud computing platform, which comprises the steps that: 1, a cloud data backup system based on a Hadoop distributed file system is constructed, and the system is physically divided into a client, a backup server and a Hadoop distributed file system cluster; 2, information of the backup server for providing a service for a native machine is stored in the client, and when backup or recovery needs to be carried out, a corresponding request is sent to the backup server; and 3, the backup server receives the request of the client and carries out backup and recovery of a file, when file uploading is recovered, the file is managed in a file splitting mode, before the file is uploaded, the file is split into small file blocks and then the file blocks are uploaded, and when the file is recovered, firstly, the file blocks of the file are downloaded, and after downloading of all the file blocks is completed, the file blocks are merged into the original file. The invention discloses a novel storage method based on the cloud computing platform, and improves file storage efficiency.

The invention discloses a distributed small file storage system based on UDP (User Datagram Protocol) and a data processing method thereof. The system comprises a first server, a second server, a third server, a fourth server and a fifth server, wherein the first server comprises an external server and a service monitoring server; the second server comprises a global naming server and a service monitoring server; the third server comprises an information index library server and a service monitoring server; the fourth server comprises a task managing server and a service monitoring server; the fifth server comprises a storing server and a service monitoring server; and the first server, the second server, the third server, the fourth server and the fifth server mutually transfer and process data through a UDP refitting protocol. The system can enhance the concurrent ability and the high-speed I/O (Input/Output) ability of the whole platform, and solve the problems of high-capacity storage and load balance by utilizing the UDP refitting protocol as a communication protocol of an interior server, and is especially suitable for the online services taking files as carriers.

The invention discloses a small file storage, query and deletion method and system. The method comprises the following steps: firstly, creating a large file and merging a small file into a small file storage area of the large file; during the query of the small file, querying the index information of the small file according to the offset of an index area and the file metadata information of the small file; obtaining the small file stored in the large file according to the relative offset, in the large file, of the small file in the small file index information. Through the method and system provided by the invention, the small file can be rapidly positioned according to the metadata information of the small file, so that the searching and processing efficiency of the small file in the large file is effectively improved.