Enhanced data compression for sparse multidimensional ordered series data

Abstract

Methods and systems for compressing sparse multidimensional ordered series data. The data may include indexed data sets events, where each data set comprises an index, and a set of second parameter variables. The methods and systems are particularly suited for compression of data recorded in double precision floating point format. In some examples, the methods are tailored for compression of analytical data, such as spectroscopic data or flow cytometry data.

Description

202515949ENHANCED DATA COMPRESSION FOR SPARSE MULTIDIMENSIONAL ORDERED SERIES DATAINCORPORATION BY REFERENCE

[0001] All publications and patent applications mentioned in this specification are herein incorporated by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference.FIELD

[0002] This invention relates to a systems and methods for compressing large data sets, and in particular large collections of floating-point measurement data, including (but not limited to) flow cytometry data, mass spectroscopy data, microscopy data, fluorescence microscopy, and histopathology data.BACKGROUND

[0003] In many fields of science and technology, it is desired to obtain information about a subject by measuring a digital signal which represents that subject. As technology progresses, the quantity and sensitivity of sensors proliferates, and the amount of data that can be obtained correspondingly increases. In particular, in many areas of science and engineering, such as biology, chemistry, astronomy, physics, geology, and flow cytometry, large quantities of multidimensional spectroscopic data are often collected and stored. For example, mass spectrometry analysis frequently generates, within a few hours, single data sets that may be up to 30 GB or greater in size. Using currently available instruments, 104mass spectrometry datasets with 106data points and 106mass intensity values could be easily acquired, resulting in a raw file size of 4 TB. As another example, current flow cytometry devices used for research and clinical analysis frequently generate, within a span of few minutes of runtime, single data sets that may be up to several GB or greater in size. Using currently available flow cytometry instruments, 1011flow cytometry datasets comprised of 106events of 102measured parameter values with 106reflected intensity values could easily be acquired in a single year, resulting in large volume of data.

[0004] The traditional approach to this problem has been to apply some form of compression to the data in order to reduce the overall data size. In many situations it is often desirable to apply a202515949form of lossless compression, the main advantage is that no information is lost during the compression, and the data can be accurately restored. However, the amount of compression that can be obtained with lossless compression is usually far lower than what is otherwise possible with lossy compression. Hence it is often acceptable to apply lossy compression in systems that can tolerate a limited amount of loss. The lossy compression in this context refers to a compression method in which the restored data, after decompression, may not be exactly the same as the original data entered for compression. As long as the absolute difference between the restored and original data - that is, the restoration error - is small enough, the situation may be acceptable in some systems. However, once loss is allowed, it is often difficult to guarantee the accuracy of the result. Therefore, lossy compression methods have been traditionally applied in situations where there is inherent redundancy in the data, for example, in image processing, where it is known the human eye is unable to observe high frequency variations in local neighborhoods, whereby filtering out the high frequency is desirable as a means for reducing the noise.

[0005] A conventional approach to this problem is to apply some type of a lossless or lossy compression, as the case may be, to the data at hand as a means for reducing the data size. There is a large body of compression algorithms in the literature, both proprietary and open-source public domain. Commonly popular lossless algorithms include GZIP, LZW, 7ZIP, BZIP2, LZ4, ZLIB, and ZSTD, to list a few. Many of the existing compression methods have been adopted for native use by the operating systems. For example, ZLIB and BZIP2 are popular with Unix / Linux systems, while GZIP and 7ZIP are popular with Microsoft Windows computers. Methods such as ZSTD, based on running multiple compression algorithms in parallel, have also gained popularity. Yet there are numerous ongoing active research efforts to develop still better lossless compression algorithms, to handle the explosion of data.

[0006] A key property of existing lossless compression methods is that they generally make no assumptions about the data to be compressed. The data can represent anything. In fact, many of the algorithms listed above treat the data as if it is comprised of a sequence of ASCII text characters. These methods generally look for the longest repeating pattern of text strings in a given window size, and run-length encode the distance to the pattern. The core algorithms behind many of these methods are based on the original 1977 Lempel Ziv algorithm and public domain variants. See e.g. U.S. Patent Nos. 4,464,650; 4,814,764; and 4,558,302; Ziv, et al.,202515949IEEE Transactions of Information theory, Vol. IT-23, No. 3, 1977; andZiv, etal. IEEE Transactions of Information Theory, Vol. IT-24, No. 5, 1978.

[0007] Since the conventional compression methods make no assumptions about the underlying data, they are generally robust, and can compress any type of data. The amount of compression, however, is variable depending on the data content. Highly redundant or more predictable data will tend to compress much more than random unpredictable data. In some situations, with particularly difficult noisy data, several of the existing methods may actually increase the amount of data by up to 2- to 3-fold, instead of strictly compressing it. Some conventional methods are designed to have a fall back option to keep such data uncompressed, instead of attempting to compress and ending up actually expanding it.

[0008] A wide variety of sensors and analyzers, e.g. radiation or x-ray detectors and spectrometers, accelerometers, mass spectrometers and flow cytometry devices collect and store data using binary and double precision floating point values, in order to retain the highest possible accuracy. Large data sets obtained with such devices often do not include repeating patterns of text strings that can be recognized and effectively compressed by utilizing the existing compression systems. Accordingly, the need arises for methods and systems capable of compressing large data sets comprised of single and double precision floating point numbers that are otherwise not amenable to compression by conventional methods.

[0009] For example, flow cytometry is a widely used method for measuring the properties of a population of cells or particles suspended in a fluid. The cells or particles are often tagged with fluorescent biomarkers, stained with natural dyes, or treated with conjugated antibodies to bond with specific biological aspects or features of the cells of interest, the antibodies carrying specific fluorescent fluorophores or quantum dots that emit light in a given wavelength when excited by light in a different, typically lower, wavelength. The fluid containing the cells or particles is injected into the flow cytometer device, and made to flow, ideally one cell or particle at a time, through a narrow channel illuminated by a high-power pulsed multi-wavelength laser beam. A portion of the light absorbed by the cells or particles, both naturally as well as by the excited fluorophores, is emitted in a band of wavelengths, and measured by an array of highly sensitive narrow band detectors. In addition, the portion of the illumination light scattered by the cells and particles, is also measured in multiple directions, usually in a vertical and horizontal direction perpendicular to the main laser beam, and possibly at additional angles.202515949

[0010] Flow cytometry is routinely used in basic research and clinical practice. Hundreds of thousands of cells or particles can be quickly examined in a short span of time. A typical dataset containing about 1 million cells can be gathered in a few minutes. For each cell, up to 100 or more measurements are taken, from the many individual narrow-band wavelength filter detectors as well as scattered light detectors. Each measurement is of a specific light intensity, in a particular wavelength or a specific scatter angle, has a dynamic range of 107, that is, at least seven orders of magnitude. Due to the large dynamic range, each intensity measurement is recorded as a double precision 64-bit floating point value according to the IEEE 754 standard format. A single dataset can easily generate hundreds of millions of double precision numbers in a few minutes. In 2024 alone, it is estimated that over a billion flow cytometry datasets were generated. To reduce the amount of data, flow cytometry vendors occasionally convert the output to single precision 32-bit floating point numbers, however this comes at a loss of accuracy, and can only provide a limited size reduction of up to 2x.

[0011] IEEE 754 is the de-facto standard used by modern computers for representing floating point numbers. However, it is well known in the art that due to the intrinsic representation, comprised of a sign, a binary exponent and a mantissa, IEEE 754 floating point numbers are not as readily compressible as regular numbers or text, especially with existing lossless compression methods such as GZIP, LZW, 7ZIP, BZIP2, LZ4, ZLIB, and ZSTD. For widely fluctuating data values such as flow cytometry data, the resulting compression is very poor, typically less than 20%. Unlike text, where simple patterns can be exploited for compression, the number 0.999999 (encoded as 0x3F7FFFEF in IEEE 754 single precision), for example, has no common byte patterns with the number 9 (encoded as 0x41100000 hex).

[0012] As the sizes of datasets increase, traditional compression techniques are not able to manage (e.g., store transport and analyze) these large datasets quickly and efficiently. Described herein are systems and methods that address these problems.SUMMARY OF THE DISCLOSURE

[0013] The methods and apparatuses (e.g., devices, systems) described herein address the problem of compressing large sets of multidimensional ordered series data, including large collections of floating-point measurement data. Although problems addressed by the methods and apparatuses described herein relate to the compression of data, specific embodiments202515949described herein may apply this in a specific manner that is tailored to the problems uniquely faced by large sets of sparse multidimensional ordered series data. As will be clear in view of the additional discussion below, sparse data means data structured such that, within a chosen representation space, only a relatively small portion of representable points are associated with meaningful signal while many values are missing, zero, near a baseline, and / or otherwise dominated by noise and / or artifacts. In this disclosure, sparseness is evidenced, for example, by (a) ordered series having relatively few significant peaks interspersed with gaps or quiet periods (e.g., mass spectrometry peak series), (b) image tiles in which comparatively fewer pixels carry strong local variation while many pixels are relatively smooth or slowly varying (e.g., histopathology or microscopy data), and / or (c) numerical values that occupy only a small fraction of the available floating point dynamic range (e.g., observed values spanning < about 10 orders of magnitude while the floating point format spans hundreds of orders of magnitude), such that the data are “sparse” in the sense of occupying a narrow subspace of the representable domain.

[0014] The methods described herein may be well suited for (but not limited to) managing (e.g., compressing) data acquired using analytical techniques (e.g., instrumental analysis) that relate to physical, chemical and / or biological properties of samples. Examples of data may include spectroscopy (e.g., mass spectroscopy), histopathology and / or flow cytometry data.

[0015] Many sensors and analyzers, including but not limited to radiation or x-ray detectors and spectrometers, accelerometers, mass spectrometers, microscopes and fluorescent microscopes, and histopathology imagers, collect and store data using binary and double precision floating point values, in order to retain the highest possible accuracy. There is a need for methods and systems capable of compressing large data sets comprised of double precision floating point numbers, and high-dynamic range, e.g., up to 64 bits-per- pixel, integer data that are otherwise not amenable to compression by existing conventional methods.

[0016] Individual data series from sparse multidimensional ordered series data may be compressed independently of the other data series. However, in many situations, including mass spectrometry and microscopy in particular, a high level of correlation may frequently exist among subsequent data series.

[0017] The methods and systems for compressing data (e.g., multidimensional time series data) described herein may have advantages over previously described compression methods,202515949including the compressing of several data series together. By compressing more than one data series at a time, e.g., concurrently or sequentially, and using relationships (e.g., correlations) between the data in different data sets, the correlated portion(s) in a first or current data series can be predicted from one or more previously processed data series, and thereby the correlated portions may be eliminated by subtracting a predicted portion from the current series data. The advantage of this approach is in significantly reducing the amount of data that has to be encoded and transmitted for a current series data. This reduction in the amount of encoded information, for each correlated current series data, provides significantly higher compression.

[0018] During decompression, the correlated portion of a current data series can be generated from the previous data series in a similar manner, and automatically added to a current decompressed data series, to restore the original content. Therefore, the correlated portion can be fully restored without requiring any additional bits in the encoded stream.

[0019] Compressing one or more data series together is not limited to compressing whole series data at a time. Each given series data can be divided into overlapping or non-overlapping local data regions, whereby a level of correlation may be computed between corresponding local regions of a current series data and one or more previous series data. In particular, these methods and systems may identify (for use herein) a level of correlation between a current local region in current series data and the same local region in one or more previous series data. For example, a set of correlated local peaks in a current local region can be predicted from a set of corresponding peaks in local regions of one or more previous series data.

[0020] Any of these methods and systems for identifying and predicting correlated local regions may apply these predictions to eliminate the correlated portion only when a significant level (e.g., a threshold level) of correlation exists between a local region of current series data and one or more previous series data. This may prevent or reduce the introduction of noise and rounding errors in the prediction and elimination steps, which are less reliable in the presence of weak correlation levels.

[0021] When correlation prediction is used in certain local regions and not in others, it may be helpful to introduce additional bits into the compressed stream, in order to direct the decoder, during decompression time, to apply correlation prediction restore or not, as the case may be, for each local region. The additional bits, e.g., per each local region, may increase the compressed file size and reduce the compression.202515949

[0022] Also described herein are methods and systems for eliminating or reducing the need for such additional bits, per each local region, without compromising the compression. The level of correlation between a current series data and one or more previous series data may be computed and compared to a given threshold. If the correlation level in a current local region is equal to or exceeds the given threshold, a prediction algorithm may be applied, to predict the correlation level and subtract it from the current series data. Otherwise, if the calculated correlation level in a current local region is below the given threshold, no prediction is applied. The process may be repeated for each local region in turn, automatically detecting the presence of, and eliminating the correlated portion, in each local region, whenever a significant level of correlation exists, without introducing any additional bits into the compressed stream.

[0023] At decompression time, the level of correlation in each local region may again be computed from the restored data. If a correlation level in a current local region is equal or exceeds the given threshold, the correlated portion of a current data series can then be generated from the previous data series in a similar manner, and automatically be added to the current decompressed data series, to restore the original content. Therefore, the correlated portion can be fully restored without requiring any additional bits in the encoded stream.

[0024] For example, these data compression methods and systems may be particularly useful for mass spectrometry (MS) data. Thus, in any of these variations, the data can include a multidimensional mass spectrometry series data, wherein the data comprises indexed data sets, each indexed data set comprising an index (n) representing the retention time index, a first variable (xn) representing the mass over charge m / z at index n, and a second variable (yn) representing the intensity or count of the number of ion fragments observed by the instrument.

[0025] In some variations the data can be multi-dimensional mass spectrometry series data, wherein the data comprise of indexed data sets, each indexed data set comprising an index (n) representing the retention time index, a first variable (xn) representing the mass over charge m / z at index n, a second variable (yn) representing the intensity or count of the number of ion fragments observed by the instrument, and a third variable (zn) representing the observed ion mobility of the ions traveling through an ion mobility drift tube.

[0026] Any other type of multi-dimensional data, including in particular multi-dimensional imaging data may be processed by these methods and systems. For example, in some variations the data can be multi-dimensional Brightfield microscopy image series data; the data may202515949comprise indexed data sets, each indexed data set comprising an index (n) representing the pixel index within a given local region of the image, and a first variable (xn) representing the pixel intensity captured by the microscope.

[0027] For example, the data can be multi-dimensional Bright-field microscopy image series data, the data comprise of indexed data sets, each indexed data set comprising an index (n) representing a pixel index within a given local region of the image, a first variable (xn) representing the intensity of a first color component, a second variable (yn) representing the intensity of a second color component, and a third variable (zn) representing the intensity of a third color component of the image. It should be noted, without any loss of generality, that any number of variables may be used, for example, any number of color components, and the components can be any color components of a defined color space, for example, standard RGB or YCbCr luminance-chrominance representation.

[0028] In some variations the data can be multi-dimensional multiplexed fluorescence microscopy series data, wherein the data comprise of indexed data sets, each indexed data set comprising an index (n) the pixel index within a given local region of the image, a second index variable (k) representing a particular fluorescence dye layer, a first variable (xn) representing the intensity of a first color component, a second variable (yn) representing the intensity of a second color component, and a third variable (zn) representing the intensity of a third color component of the image. As mentioned, any number of variables (e.g., dimensions) may be included and used, without any loss of generality. For example, with florescence microscopy data, the number of variables can be any number of color components, and the components can be any color components of a defined color space, such as standard RGB or YCbCr luminance-chrominance representation, for example.

[0029] For example, the data can be multi-dimensional multiplexed fluorescence microscopy series data, wherein the first, second, and third color components are full color images, the color of each image being a pseudo-color artificially assigned by the microscope operator in order to visualize complex cell or biomarker interactions by layering information from several multiple dyes on top of each other. For example, DAPI (4',6-diamidino-2-phenylindole) is a blue-fluorescent DNA stain which is commonly used as a nuclear counterstain in fluorescence microscopy, flow cytometry, and chromosome staining, while Alexa Four 546 is a bright yellow dye that can be optimized for various cellular labeling and detection. The observed fluorescence202515949of each dye is represented as a monochrome image, whereby the microscope operator may typically assign a particular shade of blue color, selected from a palette of colors, for DAPI and a particular shade of yellow, again selected from a palette of colors, to Alexa Four 546, and so on, in order to simultaneously visualize the interaction of multiple dyes together, by assigning a specific color to each monochrome image, and layering the pseudo-colorized images on top of each other, to form a typical multiplexed fluorescence microscopy image.

[0030] The methods and apparatuses described herein may be used, for example, with indexed data sets, each index indexed data set comprising an index (n) representing the pixel index within a given local region of the image, whereby the entire series data is divided into N overlapping or non-overlapping local regions, wherein each local region comprised of a length of subsequent pixels, and the compression applied to each local region in turn.

[0031] In some variations the data comprises indexed data sets, each indexed data set comprising an index (n) representing the pixel index within a given local region of the image, whereby the entire series data is divided into N overlapping or non- overlapping local regions, wherein each local region comprised of a rectangular local neighborhood of a certain width and height, and the compression applied to each local region in turn.

[0032] The methods and apparatuses (e.g., systems) described herein may significantly compress sparse, multidimensional ordered series data in a manner that has not previously been possible. For example, as compared to a standard file compression for mass spectrometry (MS) data, such as “ZIP” file compression, which may only provide about 2-3 fold reduction in file size, the compression methods described herein may result in 300 fold reduction in file size.

[0033] For convenience, the datasets described herein may be referred to as either prior motif compression (PM compression) or prior correlated region (PCR) compression.

[0034] For example, a computer-implemented method for compressing sparse multidimensional ordered series data may include: dividing a multidimensional ordered series data into a plurality of local regions, wherein the data in each local region comprise one or more indexed data sets, each indexed data set comprising an index (n) within a given local region of the image and one or more variables that are indexed by the index (n) identifying a predictor that calculates each of the one or more variables as a function of the index (n), a current local region data, and at least one previous local region data; adjusting the current local region data by subtracting a scaled predicted correlated data based on the at least one previous local region data202515949when a level of correlation exceeding a threshold exists between the current local region data and the previous local region data; and encoding the adjusted current local region data, including an optimum scale factor, into a compressed stream.

[0035] The multidimensional ordered series data may be any type of data, particularly imaging and / or spectrographic data. The data may be divided into a plurality of overlapping local regions, or a plurality of non-overlapping local regions.

[0036] According to one example, a computer-implemented method of compressing a sparse multidimensional ordered series data includes: accessing the sparse multidimensional ordered series data comprising indexed data sets, each indexed data set comprising a set of variables (ymn), wherein n is an event index representing an event, and m is a parameter index representing a parameter, wherein each event index (n) is associated with multiple parameter indexes (m); partitioning the sparse multidimensional ordered series data into subsets of data by: un-mixing the set of variables (ymn), to reduce cross-parameter dependencies introduced by the measurement process, thereby providing un-mixed data; classifying, using a classifier, the unmixed data based on one or more desired criteria; and splitting the un-mixed data into the subsets of data, wherein each subset of data is comprised of a set of parameter indexes (m) of one or more parameters; for each classified subset of data: predicting, using a predictor, a value for each variable (ymn) as a function of the event index (n) based on a trend observed in the classified subset of data, wherein a representative value is selected as the value when a trend is not observed in the classified subset of data; calculating a series of corrections (cmn) for each variable (ymn) from the predicted value; calculating a hop offset value (Amn) for each correction (cmn) of the series of corrections (cmn); and generating a compressed output, the compressed output comprising a decoder legend configured to decode the predictor, the series of corrections (cmn), and the hop offset value (Amn) for each classified subset of data.

[0037] Calculating the series of corrections (cmn) for each variable (ymn) may include assigning an intensity code word to each correction (cmn) based on a value and a frequency of the correction (cmn), wherein calculating the hop offset value (Amn) for each correction (cmn) may include assigning a hop code word to each hop offset value (Amn) based on the value and frequency of the hop offset value (Amn).

[0038] The compressed output further may include code word data comprising the intensity code word and the hop code word for each correction (cmn) and each hop offset value (Amn),202515949respectively. The decoder legend may include a description of the predictor, a reverse intensity code word dictionary associated with the correction (cmn), and a reverse hop code word dictionary associated with the hop offset value (Amn).

[0039] The sparse multidimensional ordered series data may be in single and / or double precision floating point format. The sparse multidimensional ordered series data may contain values that fall within a dynamic range of less than 10 orders of magnitude. The sparse multidimensional ordered series data may include multiple indexed x,y pairs.

[0040] The subsets of data may be identified by a classifier. The predictor may be a global predictor function. The global predictor may be an nthorder polynomial function. The nthorder polynomial function may be g(n) = ao +ai*n + a2*n2 + a3*ns. The predictor may be a piecewise predictor. The predictor may be a local predictor. The predictor may further include an error correction mechanism.

[0041] The sparse multidimensional ordered series data may be spectroscopic data. The sparse multidimensional ordered series data may include flow cytometry data. The parameters may include one or more of: forward scatter measurements, side scatter measurements, fluorescence intensity measurements for each fluorescence channel, and time and / or duration each cell passes through light beam.

[0042] Predicting, using the predictor, the value for each variable (ymn) as a function of the event index (n) may include: identifying a correlation among the set of parameter values (m) of a first classified subset of data having similar trends or lack of a trend as a function of the event index (n); determining a main (ymn) parameter from among the set of correlated parameter values (m), one of the set of correlated parameter values (m) or a combination of several correlated parameter values (m), according to a desired criteria; predicting, using a predictor, a value for the main parameter (ymn) as a function of the event index (n) based on a trend observed in the classified subset of data, wherein a representative value is selected as the value when a trend is not observed in the classified subset of data; determining one or more dependent correlated parameter values (m) using offsets and scaling factors associated with the identified correlation; and predicting the values of each variable (ymn) of dependent correlated parameter values (m) in classified subsets of data using the one or more offsets and scaling factor.

[0043] Predicting, using the predictor, the value for each variable (ymn) as a function of the event index (n), includes dividing a plurality of event indexes (n) into a continuous sub-sequence202515949range of the event index (n), starting at event ni and ending at event n2, providing a piecewise correlated prediction.

[0044] Any of these methods (or apparatuses for performing them) may compute the correlation level between the current local region and the previous local region. For example, for MS data, the correlation level may be computed from a series of similar peaks that exist between the current local region data and at least one previous local region data. The series subset of similar peaks may be considered similar if the majority of peaks from the subset of similar peaks have one or more of: approximately a same mass-to-charge ratio, approximately a same charge state as determined from the spacing between subsequent peaks, and similar peak intensity abundance distributions that match an averagine model.

[0045] The compressed stream may include the modified local region(s) as described above, along with the optimal scale factor (also referred to equivalently herein as an optimized scale factor or an optimum scale factor). The optimized scale factor may typically include one value per local region subset for a series of peaks (as opposed to an individual scale factor per peak value).

[0046] The previous local region (which may also be referred to herein as a prior local region or a template region) may be set by agreement between the encoder and decoder. In some variations the previous local region may be set to correspond to the previous local region in the scan order (for example, the region immediately to the left in raster-scan order). In another variation, the previous local region could be one of: (a) immediately left, (b) immediately above, or (c) immediately above and left (diagonal), as described herein. In some variations, the encoder may inspect all three, and may choose the one with the highest correlation level. The decoder may determine the previous local region from the data itself (e.g., it may inspect all three possible regions, after restoring the content) or it may be provided with the identity of the previous local region encoded within the compressed stream. In some variations the previous local region may be an average of (a), (b), and (c) above, etc. Thus, in general, the encoder and the decoder use the same referenced previous local region.

[0047] For example, the encoder and decoder may both be set to use as the previous local region, a region that is immediately to the left of the current local region, as described in (a), above. This may provide a particular compression result. In some variations, the compression may be increased by choosing one (or a combination of) the prior local region(s) immediately to202515949the left, above, and / or diagonal from the current local region, e.g., corresponding to (a), (b), or (c), above. In variations in which the encoder is looking at more than one (e.g., it may look at all three of these regions, a, b, and / or c), it may find a higher correlation in one of these regions as compared to others of these regions. As mentioned, in a given implementation, the encoder and decoder may be set to the same (e.g., fixed) method, e.g., always using one or more of these prior local regions (a, b, and / or c).

[0048] In some implementations, the header (e.g., one byte of the header) may be used to specify the particular prior local region(s) to be used, e.g., out of a predefined list of available methods, such as indicating the prior local region immediately to the left, immediately above, and / or immediately above and to the left (e.g., diagonal) from the current local region. Thus, the encoder (which may encode this header) may tell the decoder (which may read this header) which of the predefined prior local region(s) to use; for example, by sending a code in a particular header byte.

[0049] Also described herein are systems for compressing sparse multidimensional ordered series data. These systems may include a non-transitory computer-readable medium with instructions stored thereon, that when executed by a processor, cause the processor to perform any of the methods described herein. For example, the non-transitory computer-readable medium with instructions stored thereon may be configured to: divide a multidimensional ordered series data into a plurality of local regions, wherein the data in each local region comprise one or more indexed data sets, each indexed data set comprising an index (n) within a given local region of the image and one or more variables that are indexed by the index (n) identify a predictor that calculates each of the one or more variables as a function of the index (n), a current local region data, and at least one previous local region data; adjust the current local region data by subtracting a scaled predicted correlated data based on the at least one previous local region data when a level of correlation exceeding a threshold exists between the current local region data and the previous local region data; and encode the adjusted current local region data, including an optimum scale factor, into a compressed stream.

[0050] As mentioned, also described herein are methods of decompressing any of the compressed data (e.g., the sparse multidimensional ordered series data) described herein. For example, described herein are computer-implemented methods for decompressing sparse multidimensional ordered series data from a compressed data file or stream, the method202515949comprising: receiving the compressed data file or stream; decoding an adjusted local region data from the compressed data file or stream, wherein the decoded adjusted local region data corresponds to a target local region to be restored; decoding a predictor scale factor from the compressed data file or stream by comparing the adjusted local region data to one or more prior local regions to generate a predicted local region peaks that are scaled by an optimum scale factor; adding the predicted local region peaks and the decoded adjusted local region data together to provide a restored local region data; and outputting the restored local region data.

[0051] For example, a computer-implemented method for decompressing sparse multidimensional ordered series data from a compressed data file or stream may include: receiving the compressed data file or stream; decoding an adjusted local region data from the compressed data file or stream, wherein the decoded adjusted local region data corresponds to a target local region to be restored; decoding a predictor scale factor from the compressed data file or stream by comparing the adjusted local region data to one or more prior local region data that are offset from the adjusted local region data to identify a subset of peaks from the adjusted local region data and scaling the subset of peaks by an optimum scale factor extracted from the compressed data file or stream to generate a predicted local region peaks; adding the predicted local region peaks and the decoded adjusted local region data together to generate a restored local region data; and outputting the restored local region data.

[0052] Also described herein are non-transitory computer-readable media with instructions stored thereon, that when executed by a processor, cause the processor to perform any of the computer-implemented methods described herein, and / or systems including these non-transitory computer-readable media.

[0053] In any of these methods and apparatuses (e.g., systems, media, etc.) may include adjusting the local region data and the predictor scale factor using a decoder that applies the encoding used to encode the data.

[0054] According to another example, a computer-implemented method for compressing floating point multidimensional ordered series data comprises: (a) receiving floating point multidimensional ordered series data, wherein the data comprise of indexed data sets, each indexed data set comprising an index (n) of an event, and a set of second variables (yin, ... , ykn) parameter values; (b) un-mixing or de-interleaving the data; (c) defining a classifier, to identify data subsets having similar properties or range values, and splitting the original data into separate202515949data subsets, to be compressed independently; (d) when a data subset exhibits a trend, defining a predictor that calculates each second variable (ymn) as a function of the index (n), otherwise, if no trend could be identified, selecting a subset representative value based a given criteria, such as the most frequent value or average value, etc. ; (e) calculating a set of corrections (cmn) for each (ymn) from predicted or representative values for all members in each data subset; (f) generating a set of personalized code words for each data subset, independent of the other subsets, by preassigning an amplitude code word to each cmnbased on the value and frequency of the Cmn; (g) calculating a hop offset value (Amn) for each cmn; (h) assigning a hop code word to each Amnbased on the value and frequency of the Amn; and (i) generating a compressed output a decoder legend and code word data comprising an amplitude code word and a hop code word for each Cmn and each Amn, for each data subset, respectively. The compressed output comprises a description of the predictor or representative value; a compressed binary index vector for each parameter; a reverse amplitude code word dictionary associated with Cmn; and a reverse hop code word dictionary associated with Amn.

[0055] Another example provides non-transitory computer readable media having instructions stored therein, which, when executed by a process, cause the processor to perform operations, the operations comprising: receiving sparse multidimensional ordered series data, wherein the data comprise indexed data sets, each indexed data set comprising an index (n) of an event, and a set of second variables (yin, ... , ykn) parameter values; (b) un-mixing or deinterleaving the data; (c) defining a classifier, to identify data subsets having similar properties or range values, and splitting the original data into k separate data subsets, to be compressed independently; (d) when a data subset exhibits a trend, defining a predictor that calculates each second variable (ymn) as a function of the index (n), otherwise, if no trend could be identified, selecting a data subset representative value based a given criteria, such as the most frequent value or average value, etc.; (e) calculating a set of corrections (cmn) for each (ymn) from predicted or representative values for all members in each data subset; (f) generating a set of personalized code words for each data subset, independent of the other subsets, by pre-assigning an amplitude code word to each cmnbased on the value and frequency of the cmn; (g) calculating a hop offset value (Amn) for each cmn; (h) assigning a hop code word to each Amnbased on the value and frequency of the Amn; and (i) generating a compressed output a decoder legend and code word data comprising an amplitude code word and a hop code word for each cmn and each202515949Amn, for each data subset, respectively. The compressed output comprises a description of the predictor or representative value; a compressed binary index vector for each parameter; a reverse amplitude code word dictionary associated with cmn; and a reverse hop code word dictionary associated with Amn.

[0056] Yet another example provides ordered series data encoders comprising: a data receiver for receiving sparse multidimensional ordered series data, wherein the data comprise indexed data sets, each indexed data set comprising an index (n) of an event, and a set of second variables (yin, ... , ykn) parameter values; a data un-mixer; a classifier to identify data subsets having similar properties or range values; a data splitter for splitting the data into separate subsets, to be compressed independently; a predictor (e.g., predictor component) that predicts each subset variable (ymn) as a function of the index (n) or selected subset representative, such as the most frequent value or average value, etc. ; a corrector for calculating a set of corrections (cmn) for each (ymn) from predicted or selected representative values; an amplitude coder (e.g., amplitude code component) that assigns a personalized amplitude code word to each cmnbased on the value and frequency of the Cmn; a hop coder (e.g., hop code component) that calculates a hop value (Amn) for each Cmn and assigns a hop code word to each Amnbased on the value and frequency of the Amn; and an encoder (e.g., encoder component) that generates a compressed output, said compressed output comprising a decoder legend and code word data comprising an amplitude code word and a hop code word for each cmnand each Amn, respectively. The decoder legend comprises a description of the predictor; a reverse amplitude code word dictionary associated with ymn; and a reverse hop code word dictionary associated with Amn. In a preferred implementation, the ordered series is a time series or an event series.

[0057] The data of interest may be collected by a measurement process for measuring one or more parameters. For example, the data may be collected using a set of sensors, detectors, and / or a variety of instruments and devices. The devices can vary from common sensors such as velocity, accelerometers, and location positioning systems found in any smart phone to highly specialized instruments such as flow cytometers and radiation or x-ray detectors. For flow cytometry measurements, the parameters may include, for example, forward scatter measurements, side scatter measurements, fluorescence intensity measurements for each fluorescence channel, at time that each cell passes through light beam and / or duration of each cell that passes through light beam. Regardless of the type of sensing device, the data is typically202515949collected and stored using double precision floating point numbers, in order to retain the highest possible accuracy.

[0058] In some implementations, the multidimensional ordered series data is in double precision floating point format. Preferably, the multidimensional ordered series data contain values that fall within a dynamic range of less than 10 orders of magnitude. In many implementations, the multidimensional ordered series data, preferably time series data or event data, comprises a plurality of indexed parameter values. In a preferred implementation, the multidimensional ordered series data is flow cytometry data, wherein ymnrepresents the measured light intensity value of parameter index m, one of many parameters, observed at index (n) of an event.

[0059] A high level of compression can be achieved when the data representation of one or more ymnparameter is sparse. In some implementations, the values of a parameter may exhibit a trend, where the parameter data is a smooth and slowly varying function of the index, for example increasing or decreasing with the index (n). A low-order polynomial predictor can be used to model and accurately predict the observed ymnparameter behavior. While defining the predictor, the predictor can be a global predictor function, a piecewise predictor, or a local predictor. The predictor can, optionally, comprise an error correction mechanism. Preferably when the predictor is a global predictor function, the predictor is an nthorder polynomial function. In a preferred implementation, the predictor is the function, g(n) = ao +ai*n + a2*n2 + a3*ns, wherein ao, ai, a2 and as are constants and n is the index.

[0060] A high level of compression can be achieved when the data representation of one or more ymnparameters is sparse. In some implementations, the values of a parameter may comprise a sequence of variable amplitude measurements interspaced with intervals of relatively quiet periods during which the values remains moderately constant and primarily dominated by noise. Preferably, the ymnparameter data is comprised of a multi-modal distribution of amplitude values, where certain amplitude ranges that occur frequently are interspaced with other amplitude ranges that occur much less frequently. In some cases, the parameter data preferably is comprised of a discrete set of observable amplitude ranges interspaced with intervals of amplitude ranges that are not observed in the data. In a preferred implementation, the amplitude ranges are light intensity values or corrections cmnfrom predicted light intensity values.202515949

[0061] In certain implementations, the step of assigning an amplitude code word to each cmnbased on the value and frequency of the cmncomprises: (i) generating a hash table for amplitude values; and (ii) looking up each of the (Cmn) values in turn; (iii) sorting the list of amplitude values by their associated frequency occurrence; and (iv) assigning a personalized unique amplitude code word to each unique amplitude value in the list of amplitude values, wherein the shortest code words are assigned to the most frequently occurring amplitude values. When looking up each of the cmnvalues, if a cmnvalue is not previously seen, then the cmnvalue is added to a list of amplitude values and an associated frequency occurrence is set to one. And if the Cmn is already present on the list of amplitude values, then the associated frequency occurrence is incremented by one. In a preferred implementation, any cmnvalue less than or equal to a baseline threshold is skipped. In some implementations, assigning an amplitude code word to each Cmn can include assigning a single code word to represent two or more cmnvalues.

[0062] In some implementations, the sparse multidimensional ordered series data describe a multi-modal distribution of hop Amnranges, where certain hop ranges that are frequently and considerably more likely to occur are interspaced with other hop ranges that are much less likely to occur. In yet other (non-mutually exclusive) implementations, the hop offset values are comprised of a discrete set of observable amplitude ranges interspaced with intervals of amplitude ranges that are not observed in the data.

[0063] In certain implementations, the step of calculating a hop offset value (Amn) for each Cmn comprises: (i) identifying an initial hop offset value (A0) and entering the A0 into a previous register as a previous index location; (ii) feeding each index (n) into the previous register subtracting the previous index location from the index (n) to calculate the hop offset value (Amn) and then replacing the previous index location with the index (n); and (iii) repeating step (ii) for each index (n) in the sparse multidimensional ordered series data. In an alternate implementation, the step of calculating a hop offset value (Amn) for each cmncomprises: (i) identifying an initial hop offset value (A0) and entering the A0 into a previous register as a previous index location; (ii) feeding each cmnvalue, in turn, into the previous register subtracting the previous index location from the cmnvalue to calculate the hop offset value (Amn) and then replacing the previous index location with the current cmnvalue; and (iii) repeating step (ii) for each Cmn in the sparse multidimensional ordered series data.202515949

[0064] In one implementation, the step of assigning a hop code word to each Amnbased on the value and frequency of the Amncomprises: (i) generating a hash table for hop offset values; (ii) looking up each hop offset value (Amn) value in turn; (iii) sorting the list of hop values by their associated frequency occurrence; and (iv) assigning a unique hop code word to each unique hop value in the list of hop values, wherein the shortest code words are assigned to the most frequently occurring hop values. When looking up each hop offset value, if a Amnvalue is not previously seen, then the Amnvalue is added to a list of hop values and an associated frequency occurrence is set to one, and if the Amnis already present on the list of hop values, then the associated frequency occurrence is incremented by one. In some implementations, assigning a hop code word to each Amncan include assigning a single hope code word to represent two or more Amnvalues.

[0065] According to yet another example, a non-transitory readable media comprises compressed sparse multidimensional ordered series data, the compressed data comprising: a decoder legend and code word data comprising a plurality of pairs of an amplitude code word and a hop code word, wherein each pair of an amplitude code word and a hop code word, is capable of being decompressed to an index (n), and a plurality of second variables (ymn). The decoder legend comprises a description of a predictor, wherein the predictor calculates a set of predicted variable (ymn) values as a function of an index (n); a reverse amplitude dictionary, wherein the reverse amplitude dictionary includes a plurality of amplitude code words, wherein each amplitude code word is associated with an amplitude value; and a reverse hop offset dictionary, wherein the reverse hop offset dictionary associates a hop code word to each Amnbased on the value and frequency of the Amn. Preferably, the decoder legend further comprises an initial hop offset value (AO).

[0066] In general, the methods and apparatuses (e.g., software, firmware, hardware) described herein may be specifically tailored to the compression of sparse multidimensional ordered series of flow cytometry spectroscopic data. In particular, these compression techniques described herein may include components, such as predictors, encoders, and the like, that are specific for use with expected formats and values of flow cytometry data. For example, described herein are flow cytometry machines that have been modified to include software, hardware and / or firmware for encoding the large volumes of flow cytometry spectroscopy data recorded using the compression described herein. Such machines may therefore operate more202515949efficiently, accurately, quickly and for a potentially longer time due to the compression data size reduction, and may transmit and save, or even operate directly on, the resulting highly compressed data. For example, described herein are flow cytometry data collection and / or analysis devices including a processor and a non-transitory computer readable medium having instructions stored therein, which, when executed by the process, cause the processor to: receive sparse multidimensional ordered series data, wherein the data comprise indexed data sets, each indexed data set comprising an index (n) of an event, and a set of second variables (yin, ... , ykn) parameter values (which may represent observed light intensity values from a plurality of narrow wavelength filter and scatter detectors); (b) un-mixing or de-interleaving the data; (c) define a classifier, to identify data subsets having similar properties or range values, and split the original data into separate data subsets, for independent processing; (d) when a subset exhibits a trend, define a predictor that calculates each second variable (ymn) as a function of the index (n), otherwise, if no trend could be identified, select a subset representative value based a given criteria, such as the most frequent value or average value, etc.; (e) calculate a set of corrections (cmn) for each (ymn) from predicted or representative values for all members in each data subset; (f) generate a set of personalized code words for each data subset, independent of the other subsets, by pre-assigning an amplitude code word to each Cmn based on the value and frequency of the Cmn; (g) calculating a hop offset value (Amn) for each cmn; (h) assigning a hop code word to each Amn based on the value and frequency of the Amn; and (i) generating a compressed output a decoder legend and code word data comprising an amplitude code word and a hop code word for each Cmn and each Amn, for each data subset, respectively. The compressed output comprises a description of the predictor or representative value; a compressed binary index vector for each parameter; a reverse amplitude code word dictionary associated with cmn; and a reverse hop code word dictionary associated with Amn, respectively.

[0067] All of the methods and apparatuses described herein, in any combination, are herein contemplated and can be used to achieve the benefits as described herein.

[0068] These and other aspects and details are described herein.202515949BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Novel features of embodiments described herein are set forth with particularity in the appended claims. A better understanding of the features and advantages of the embodiments may be obtained by reference to the following detailed description that sets forth illustrative embodiments and the accompanying drawings.

[0070] FIG. 1 shows one example of a block diagram illustrating one example of a multidimensional data compression method.

[0071] FIG. 2 illustrates one example of MS data that may be compressed using the methods described herein. In FIG. 2, the MS data shows a current local region data, showing a set of indexed series data each comprised of an index (n), a first variable (xn) (in this example, the mass-to-charge ratio, m / z), and a second variable (yn) (in this example, the intensity abundance count of ion fragments with the given m / z). The horizontal axis is mass-to-charge ratio, and vertical axis is the intensity ion fragment count.

[0072] FIG. 3 shows an example of corresponding previous local region data, showing a set of indexed series data, each comprised of an index (n), a first variable (xn) (mass-to-charge ratio, m / z), and a second variable (yn) (intensity abundance count of ion fragments with the given m / z).

[0073] FIG. 4 shows an example of a set of correlated data between a current local region data from FIG. 2 and a particular previous local region data in FIG. 3. Only some of the data, some of the peaks in FIG. 2, correlate with some of the peaks in FIG. 3. Other peaks in FIGS. 2 and 3 are not correlated. In general, correlated peaks must have certain properties in common, such as: almost the same mass-to-charge ratio m / z value, within the instrument accuracy; almost equal spacing between subsequent peaks, confirming a certain charge state; and the observed intensity distribution should generally follow the averaging model. Note the data from FIG. 3 is drawn slightly shifted away relative to the data from FIG. 2 only for illustration purpose, in practice the peaks will fall almost on top of each other, within the instrument m / z accuracy of about 1 parts-per-million typical.

[0074] FIG. 5 shows an example of an adjusted current local region data, after subtraction of predicted correlated data in FIG. 4. The adjusted data comprised of residues (circled), after subtraction of predicted correlated previous peaks (FIG. 3) from original current local region peaks (FIG. 2) as shown in FIG 4. The uncorrelated current local region peaks (on either side)202515949are not adjusted. The result of the adjustment is the replacement of high- intensity peaks (as in FIG. 2) with much smaller residues 405, as highlighted by dotted line 450, and as can be seen by comparing FIGS. 2 and 5. The adjusted data is encoded into the compressed stream.

[0075] FIG. 6 shows one example of a three-dimensional (3D) elution profile of a typical mass spectrometry data over time. Each ion fragment exhibits a typical bell-shaped curve. The intensity of each curve is low at the beginning, then gradually rises to reach an apex, then gradually goes down to zero. The time duration and apex intensity height can vary based on the fragment composition. There may be some overlap between co-eluting ion fragments on rout to the detector.

[0076] FIG. 7 shows the observed two-dimensional elution profile of the example “typical” mass spectrometry data in FIG. 6, after integration over the mass-to-charge ratio m / z axis, over time, also known as Total Ion Current (TIC). The total ion current is proportional to the elution profile envelope, the maximum of the individual curves at any given time. Since ions are electrically charged, a direct measurement of the detector current is proportional to the elution envelop. The minima points of the elution envelope define the cross-over transition time-points t2, t3, t4 from one ion fragment species to another.

[0077] FIG. 8A illustrates one example of a small region of an image forming a histopathology data set shown unprocessed (having a very large file size, e.g., 1.764 GB).

[0078] FIG. 8B illustrates the data set of FIG. 8A compressed and restored using the method described herein.

[0079] FIG. 9 schematically illustrates a block diagram of one example of a method of restoring (e.g., de-compressing) all or a portion of a multi-dimensional data set, as described herein.

[0080] FIG. 10 illustrates the direct access of a portion of a multi-dimensional data set in the compressed stream using the methods and / or apparatuses described herein.

[0081] FIGS. 11 A-l IB show another example of a dataset that may be compressed as described herein. FIG. 11 A shows an image representing the entire dataset, while FIG. 1 IB shows an enlarged view of a sub-region (region “B”) of the dataset.

[0082] FIG. 12A illustrates one example of an uncompressed multi-dimensional data set, corresponding to a multi-dimensional Bright-field microscopy image data.

[0083] FIGS. 12B-12C compare the quality of compression using the methods as described herein and traditional JPEG compression. FIG. 12B shows a magnified portion of the multi-202515949dimensional Bright-field microscopy image data of FIG. 12A (region Z) that has been compressed and restored using the methods described herein. FIG. 12C shows a magnified image of the same region of FIG. 12A that has been compressed and restored using JPEG compression at a high compression ratio.

[0084] FIG. 13 is an example diagram illustrating flow cytometry principles. The cells or particles to be analyzed, suspended in liquid, are flowing past a high-power laser beam. A portion of the light is absorbed by the cell or particle, exciting various dyes and reagents applied as treatment to the cells, and light is re-emitted at specific higher wavelengths. The emitted light is routed through a sequence of successive semi-transparent mirrors to a parallel set of narrowwavelength filters and detectors. In addition, the portion of light reflected and scattered from a cell or particle is also measured by a second parallel set of scatter detectors. For each event, namely a cell or particle detected to be present in the light beam path, all parallel detectors are triggered to take light intensity measurements at the same time, and save the data to file, as a sequence of floating point values. Depending on the experiment, there can be up to 100 or more detectors overall. In a short span of a few minutes, millions of cells or particles can flow through the light beam, resulting in large amounts of gathered data. Because of the high flow rate, there is no time to process the data. However, once saved, a need for compression exists, to reduce the data size.

[0085] FIG. 14 is graph showing an example of how light emitted by dyes or fluorophores from cells in flow cytometry are emitted in ranges of wavelengths, rather than a single pure wavelength, illustrating how this series data includes mixed data. Light from the dyes or fluorophores are emitted from each detector channel (Chi, Ch2, Ch3) with a characteristic bellshaped intensity profile and a peak intensity at a specified wavelength. In this example, three dyes or fluorophores attached to treated cells react to the laser excitation light. The third detector channel (Ch3) detects light emitted from a third dye with a peak intensity around 460 nm, but as shown, it is emitting light in a range of wavelengths, from 380 to 640 nm. Likewise, the second detector channel (Ch2) detects light emitted from a second dye that emits light from 470 to 660 nm, peaking at 540 nm; and the first detector channel (Chi) detects light emitted from a first dye that emits light from 570 to 700 nm, peaking at 595 nm. In real flow cytometry data, there can be many dye / fluorophore signals.

[0086] FIG. 15 provides an example of one typical intensity output of one of the parallel narrow-202515949wavelength filter detectors. The cells in this case have been treated with a particular dye stain known as Live / Dead cell indicator, or cell viability test. This particular dye is excited by light at a wavelength of 488 nm (nanometer), and emits light at around 617 nm. The fluorescent dye is reactive with inter cellular proteins, but cannot penetrate live cell membranes. In the presence of live cells (FIG. 15(a)), it can only react with few exterior surface proteins, resulting in low intensity values. In presence of dead cells (FIG. 15(b)), on the other hand, the dye can penetrate the damaged membranes of dead cells, and react with both the exterior and interior proteins, resulting in significantly higher intensity values. The difference in fluorescence intensity between live and dead cells is typically larger than 50-fold, therefore providing complete simultaneous discrimination between live and dead cell populations. FIG. 15(c) illustrates the observed separation between the two populations. The live cells intensities are clustered around 102 intensity units, while the observed intensities of dead cells are clustered above 104 units.

[0087] FIG. 16 illustrates an example of IEEE 75464-bit double precision representation of a decimal number. First, if an exact match does not exist, a decimal number has to be rounded to the closest available representation within the IEEE 75464-bit double precision floating point standard. The closest available representation is the most accurate number, the one that minimizes the error to the decimal number value. The closest most accurate representation is expressed in terms of three distinct parts, of fixed widths: a 1 -bit sign, an 11 -bit exponent (with constant offset), and a 52-bit mantissa. The three parts are combined, to form a single 64-bit quantity. As demonstrated in FIG. 16, the rounding of decimal numbers to the closest available representation can lead to the introduction of a sequence of additional least significant decimal digits. While this rounding error is small and generally of no practical significance, it has a large impact in terms of the compression, in particular for lossless compression, that must exactly preserve all digits, including any trailing digits, in entirety, in order to be able to accurately reproduce the closest representation upon decompression, bit-for-bit, with no loss.

[0088] FIG. 17 illustrates an example of the dynamic range of the IEEE 754 industry standard 64-bit double precision floating point numbers (dotted area 1701), not drawn to scale. The observed signal range of a typical ordered series flow cytometry data series spanning the range of 10'5to 10+7intensity units is highlighted (horizontal lines 1702). While negative numbers are not natively produced by the array of narrow wavelengths filter and scatter detectors, negative numbers may still occur due to certain calculations such as the subtraction of202515949background scattered light as means to compensate for reflections of light from the channel walls and optics imperfections. In many cases, negative numbers are ignored, truncated to zero. The range of negative values is generally smaller than the corresponding range of positive values. If not truncated, a second region of negative values may be present, spanning a negative range of 10'4to 10+6intensity units (vertical lines 1703). Overall, the range of both positive and negative flow cytometry data (1702 and 1703 combined) is just a tiny fraction of the IEEE 754 double precision range (1701), occupying a small portion of the entire vast space of all possible numbers, a sparse representation.

[0089] FIG. 18 illustrates an example of the dynamic range of the IEEE 754 industry standard 32-bit single precision floating point numbers (dotted area 1801), not drawn to scale. The observed signal range of a typical ordered series flow cytometry data series spanning the range of 10'5to 10+7intensity units is highlighted (horizontal lines 1802), and a second region of negative values, if present, spanning the negative range of 10'4to 10+6intensity units (vertical lines 1803).

[0090] FIG. 19 shows a block diagram of an example method of compressing sparse multidimensional ordered series data. The input data, which may be comprised of multiple (e.g., millions) of indexed events with multiple (e.g., up to 100 or more) measurement values for each event, each measurement saved in standard IEEE 75464-bit double precision floating point representation. The interleaved input data is un-mixed, and a classifier is used to identify subsets of data having similar intensity ranges or other properties, based on given criteria. The classifier outcome is used to split the data into multiple subsets to be compressed independently. Each data subset is checked, and if determined to have a trend, a predictor is fitted to the data; otherwise, if no trend is observed, a representative value, such as the most frequent value or the average value, is selected as a subset representative. The predictor coefficients or subset representative are encoded in the compressed bit stream. Only the corrections from predicted data values need to be encoded in the compressed stream. A personalized set of code words is assigned to each data subset, for optimizing the compression. When the prediction is reasonably good, only a set of small corrections need to be sent in the compressed stream for each data subset, resulting in higher compression.202515949DETAILED DESCRIPTION

[0091] Described herein are apparatuses (e.g., devices, systems) and computer-implemented methods for compressing sparse multidimensional ordered series data, for example, analytical data. The large sets of multidimensional ordered series data may include large collections of floating-point measurement data.

[0092] In general, a goal in compressing large amounts of data is to remove noise and repeating information, to reduce the data size, while preserving necessary information for downstream operations, such that the compressed data, when decompressed, is essentially as good as the original data. The methods and apparatuses described herein may achieve high compression ratios without compromising the detail of the data, including avoiding high-compression blocking artifacts, and / or quantization. These methods and apparatuses may be used with up to 64-bit dynamic range or greater. These methods and apparatuses may also be used for fast, e.g., on-the-fly (or regional) decompression, and may allow streaming of compressed images, including as the images are being taken.

[0093] Although the methods and apparatuses described herein can be used to compress different types of data, they are well-suited for compressing sparse multidimensional ordered series data. In general, sparse multidimensional ordered series data includes data that is sparse (e.g., most of the values are zero or missing), multidimensional (e.g., includes multiple axes or features) and ordered (e.g., includes a natural sequence or order). Examples of such data may include data acquired using analytical techniques such as mass spectroscopy, histopathology imaging and flow cytometry.

[0094] As described herein, different types of sparse multidimensional ordered series data may be compressed using different methods based on characteristics of the data. For example, some methods may be applied to mixed data. In general, mixed data may refer to data that includes multiple types of data, typically of combination of numerical and / or categorical data. For example, flow cytometry data may include mixed data caused by overlapping emissions of multiple dyes within the narrow- wavelength detection bands of detectors, as described herein. Mass spectroscopy data may not include mixed data since the measured parameters (e.g., time-of-flight and signal intensity) are typically independent of one another. Some of the methods applied to processing (e.g., compressing) mixed data may not necessarily be applied to processing (e.g., compressing) data that is not mixed. However, in some cases, the same methods202515949may be applied to mixed data and data that is not mixed.

[0095] FIG. 1 shows a block diagram of one example of the compression method as described herein. The multi-dimensional input data 205 is comprised of an ordered series data, wherein the data comprise of indexed data sets, each indexed data set comprising an index ( ), at least one variable, a first variable, (xn); an optional second variables (yn); an optional third variables (zn); and so on. The number of optional variables may be determined by an input dimensionality.

[0096] In one example, the data consist of multi-dimensional mass spectrometry data, wherein the data comprise of indexed data sets, each indexed data set comprising an index ( ), representing an index in a current local region, a first variable (xn) representing the observed mass-to-charge ratio m / z, and a second variable (yn) representing the observed intensity or count of ion fragment abundance observed by the instrument. In another example, the data consist of multi-dimensional mass spectrometry series data, wherein the data comprise of indexed data sets, each index indexed data set comprising an index (n), representing an index in a current local region, a first variable (xn) representing the observed mass-to-charge ratio m / z, a second variable (yn) representing the observed intensity or count of ion fragment abundance observed by the instrument, and a third variable (z„) representing the observed ion mobility of the ions traveling through an ion mobility drift tube.

[0097] In yet another example, the data consist of multi-dimensional Bright-field microscopy image data, or Whole Slide microscopy imaging data, in which the data comprises indexed data sets, each indexed data set comprising an index (ri) of an image pixel within a current local region of the image, and a first variable (xn) representing the observed intensity of an image pixel captured by the microscope. In yet another example, the data consist of multi-dimensional Bright-field microscopy image series data, the data comprise of indexed data sets, each indexed data set comprising an index (ri) of an image pixel within a current local region of the image, a first variable (xn) representing the intensity of a first color component, a second variable (yn) representing the intensity of a second color component, and a third variable (z„) representing the intensity of a third color component of the image. It should be noted, without any loss of generality, that the number of variables can be any number of color components, and the components can be any color components of a defined color space, such as standard RGB or YCbCr luminance-chrominance representation, for example.202515949

[0098] In yet another example, the data consists of an image stack of monochrome microscopy image data, wherein the data comprised of indexed data sets, each indexed data set comprising an index (n) of an image pixel within a current local region of the image, and a set of variables (xn, yn, zn), whereby each variable represents the observed intensity of an image pixel at a particular image layer of the image stack. For example, a piece of tissue may be sliced into thin sections along a particular direction, about 1 to 5 micrometers apart. Each thin section is treated and processed separately, and the resulting images are stacked on top of each other as image layers, to provide visualization of the entire tissue. The common name of the technique is known as Z-stack.

[0099] In yet another example, the data consist of an image stack of color microscopy image data, wherein the data comprised of indexed data sets, each indexed data set comprising an index (n) of an image pixel within a current local region of the image, and a set of first, second and third variables (xn, yn, zn), whereby each first, second and third variable in the set represents an observed color intensity of image pixel at a particular image layer of an image stack, or Z-stack. It should be noted, without any loss of generality, that the number of variables can be any number of color components, and the components are the color components of a defined color space, such as standard RGB or YCbCr luminance-chrominance representation, for example.

[0100] In yet another example, the input data consist of a time sequence of a monochrome microscopy image data, wherein the data comprised of indexed data sets, each indexed data set comprising an index (n) of an image pixel within a current local region of the image, and a set of first variables (xn, yn, zn, ...), wherein each variable represents the observed intensity of image pixel at a particular time point of the sequence. For example, a piece of tissue may be monitored for a level of drug uptake or absorbance over time. A sequence of microscopy images are taken over time, and the resulting images are layered sequentially in time, like frames in a movie. The viewer can then use a conventional video player to watch the image sequence play out over time.

[0101] In one example the input data consist of a time sequence of a color microscopy image data, wherein the data comprised of indexed data sets, each indexed data set comprising an index (n) of an image pixel within a current local region of the image, and a set of first, second and third variables (xn, yn, zn), whereby each first, second and third variable in a set represents the observed of a first, second, and third color component intensity of image pixel at a particular time point of the sequence. It should be noted, without any loss of generality, that the number of202515949variables can be any number of color components, and the components can be any color components of a defined color space, such as (for example) standard RGB or YCbCr luminancechrominance representation.

[0102] In another example the data consist of a multiplexed multi-channel color fluorescent microscopy data, wherein the data comprised of indexed data sets, each indexed data set comprising an index (n), of an image pixel within a current local region of the image, and at least one or more variables (xn, yn, zn, ... ), whereby each variable represents the observed intensity of a particular fluorescent dye or stain at image pixel location. The number of variables reflects the number of different channels or dyes being used. Certain multiplexed fluorescence microscopy images may involve multiple dyes, up to ten or more, using a combination of special dyes and stains of different colors, each intended to identify the cell part boundaries, drug level concentration, internal biological state, or a positive biological marker.

[0103] In another example the data consist of multi-dimensional multiplexed fluorescence microscopy series data, wherein the data comprise of indexed data sets, each indexed data set comprising an index (n), of an image pixel within a current local region of the image, a second index variable (k) representing a particular fluorescence dye layer, a first variable (xn) representing the intensity of a first color component, a second variable (yn) representing the intensity of a second color component, and a third variable (z„) representing the intensity of a third color component of the image. It should be noted, without any loss of generality, that the number of variables can be any number of color components, and the components can be any color components of a defined color space, such (for example) as standard RGB or YCbCr luminance-chrominance representation.

[0104] In yet another example the data consist of multi-dimensional multiplexed fluorescence microscopy series data, wherein the first, second, and third color components are full color images, the color of each image artificially selected by the microscope operator from a palette of colors, in order to visualize complex cell or biomarker interactions by layering information from several multiple dyes on top of each other. For example, DAPI (4',6-diamidino-2-phenylindole) is a blue fluorescent DNA stain, commonly used as a nuclear stain in fluorescence microscopy, flow cytometry, and chromosome staining, while Alexa Four 546 is a bright yellow dye that can be optimized for various cellular labeling and detection. The observed fluorescence of each dye is captured as a monochrome image, whereby the microscope operator202515949may assign a particular color to each monochrome image, for example a particular blue color, selected from a palette of colors, for DAPI, and a particular yellow color, again selected from a palette of colors, to Alexa Four 546, and so on, in order to simultaneously view the combined interaction of multiple dyes, by assigning specific colors to each monochrome image, and layering the pseudo-colored images on top of each other, to form a typical multiplexed fluorescence microscopy image.

[0105] The data may consist of indexed data sets, each index indexed data set comprising an index (n) representing the pixel index within a given local region of the image, whereby the entire series data is divided into N overlapping or non-overlapping local regions, wherein each local region comprised of a length of subsequent pixels, and the compression applied to each local region in turn.

[0106] In some variations, the data may consist of indexed data sets, each index indexed data set comprising an index (n) representing the pixel index within a given local region of the image, whereby the entire series data is divided into N overlapping or non- overlapping local regions, wherein each local region comprised of a rectangular local neighborhood of a certain width and height, and the compression applied to each local region in turn.

[0107] The multi-dimensional input data may be divided into a set of N overlapping or nonoverlapping local regions, which may be processed sequentially, in raster scan order, one local region at a time. The local regions may be defined in terms of a localized neighborhood of indexed data sets, such as a sequential group of indexed set, or a rectangular region of indexed set, all members of a local region being in close proximity of each other.

[0108] A block diagram of one example of the compression method described herein is shown in FIG. 1. FIG. 1 shows a block diagram that illustrates just one example of a proposed multi-dimensional data compression system confirmed to perform at least one variation of the methods described herein. In FIG 1, a current local region of data 210, which may include indexed data sets, where each indexed data set comprises an index (ri), a first variable (xn), an optional additional variations (e.g., a second variable (yn), and a third variable (z„), etc.) is received from an instrument input data 205. The current local region 210 data may be the current piece of data to be encoded into the compressed stream.

[0109] When input data 205 comprise of a Z-stack or time-sequence data, the input data contains multiple layers of data, each layer corresponding to a particular tissue slice image layer202515949in a Z-stack, or a particular time point image layer of a time sequence data. The double rectangle notation in FIG. 1 indicates the presence of multiple layers in the data. While shown having a one (or two) layers, this is merely illustrative. It should be understood that the data can include any number of data layers.

[0110] Once the current local region data 210 has been processed and encoded, the local region data may be retained in a memory (e.g., in a circular buffer) for later use, e.g., as a previous local region data) and eventually discarded when no longer needed. The data of one or more previous local regions 220, closest in distance to the current local region 210, may be retrieved from the memory (e.g., circular buffer). As used herein, “closest in distance to the current local region” means a prior local region that has already been processed and stored (e.g., in a circular buffer) and that minimizes a predefined distance metric relative to the current local region, where the distance metric is defined over the ordering of the multidimensional ordered series data. In implementations in which the ordered series data are arranged as sequential scans partitioned into local regions spanning a common first-variable range (e.g., mass-to-charge, “closest in distance” comprises the corresponding local region from a most recent prior scan that spans a same first- variable range as the current local region, such that the scan-index separation between the prior scan and the current scan is minimized. In implementations in which the ordered series data are arranged as a two-dimensional grid of local regions (e.g., rectangular image tiles processed in raster order), “closest in distance” comprises a previously processed spatially adjacent local region sharing a boundary or corner with the current local region, including a local region immediately to the left, immediately above, or immediately above and to the left of the current local region, and, when multiple candidates are available, the candidate having the smallest spatial offset from the current local region according to the distance metric. For example, this could be a corresponding local region data of one or more previous scans, having the same mass-to-charge ratio (first variable xn) range as the current local region range. Or it could be an adjacent rectangular local region data, e.g., immediately to the left of a current local region, immediately above a current local region, or immediately above and to the left of a current region. All previous data may be held available in the memory (e.g., circular buffer), and can easily be accessed on a local region by local region basis.

[0111] This method may then compute the level of correlation 230 between a current local region data 210 and a previous one or more local region data 220, by comparing peak locations202515949and intensities, looking for one or more series subsets of high intensity isotope patterns, each corresponding to a subset of almost uniformly spaced peaks, with an intensity distribution that follows a general averagine model, as disclosed in more detail below.

[0112] When more than one previous local region (“prior local region”) is used, the data of several previous local regions may be averaged together, to improve the signal to noise ratio in case of noisy data. Alternatively, the level of correlation between each previous local region may be individually computed, to determine which of several previous local regions has the highest correlation with a current local region data, and this information then subsequently used to select one previous region with the highest correlation, or another metric, as the preferred previous local region to use.

[0113] For example, any of these methods or apparatuses may include, for each current local region, individually calculating the level of correlation for three previous local regions: (1) to the left of the current local region, (2) immediately above the current local region, and (3) immediately above and to the left of the current region. One of the three previous local regions, the one with the highest correlation to the current local region, may then be selected as the previous local region to use in all subsequent processing.

[0114] The computed level of correlation 230 between a current and previous local region data may be compared 250 to a pre-defined threshold 240. If the level of correlation 230 is higher or equal to the threshold 240, a predictor 260 is enabled. Otherwise, if the level of correlation 230 is lower than the threshold 240, the predictor 260 is not used, and multiplexer 270 output is forced to zero. The threshold 240 may be a predetermined correlation criterion used to decide whether predictor-based adjustment of the current local region data is permitted. In general, the threshold 240 is selected such that, when the computed correlation level 230 meets or exceeds the threshold 240, the predictor 260 is empirically expected to produce an adjusted current local region whose residuals are statistically smaller in magnitude and / or more compressible than the original current local region data, while maintaining stable and deterministic decompression. Conversely, when the computed correlation level 230 is below the threshold 240, predictor-based adjustment may be constrained to avoid injecting poorly-matched predicted content (e.g., noise-like residues) that would reduce compression performance and / or increase reconstruction error in lossy implementations. The threshold 240 may be fixed by agreement between an encoder and a decoder (e.g., as a default constant for a file format, profile,202515949or codec version), may be selected per data type (e.g., mass spectrometry versus microscopy), may be selected per channel or variable (e.g., x, y, z), and / or may be selected per local region size or shape. In some implementations, the threshold 240 may be provided in metadata (e.g., a header field) so that the decoder applies a same criterion as the encoder. In other implementations, the threshold 240 is not explicitly encoded and is instead deterministically derived at both encoding and decoding from one or more parameters that are already available to both sides (e.g., local region dimensions, bit depth, a declared instrument accuracy, or a declared compression profile identifier).

[0115] The predictor 260, when enabled, attempts to predict a correlated portion of a local region data based on the previous local region data. The correlated portion of the previous local region data is scaled with an optimum scale factor to match as close as possible the corresponding correlated portion of the current local region data.

[0116] For example, the correlated peaks of the previous local region data may each be scaled by the same factor. The predictor 260 output is passed through the multiplexer 270, the multiplexer enabled when the predictor is enabled, to the summing node 280. The optimum scaler value is also encoded into the compressed stream, one scale factor value per region.

[0117] When the computed level of correlation 230 is higher than or equal to the threshold 240, the current local region data is adjusted, by subtracting the predictor 260 output 265, through summing node 280, and the result is encoded into the compressed stream, instead of the original local region data. This adjustment essentially removes the correlated portion of the data, as best can be predicted from the previous local region data. Since the previous and current local regions are in close proximity of each other, they frequently contain similar data, which can be reliably predicted most of the time. After adjustment, only the residues of imperfect prediction remain in the data. The residues can be several orders of magnitude smaller than the original data.

[0118] A key advantage of the proposed method of the methods and apparatuses described herein over existing patents is that in existing techniques (see, e.g., US 7397961, US 8,023,750), each local region may be independently compressed. The correlated portion of each local region is therefore encoded over and over again, in each local region in turn. This consumes a lot of repeatable bits in the compressed stream. In contrast, the methods and apparatuses described herein encodes the correlated portion of the data only once, upon the first local region to be202515949encountered. From this point forward, there is no need to send the correlated portion again, only the residue differences from predicted portions are encoded in the compressed stream. This provides a substantial reduction in the amount of information to be encoded per local region, thereby delivering superior compression results.

[0119] When the computed level of correlation 230 is lower than the threshold 240, the content of the previous local region may not be as similar to a current local region data, the quality of prediction may begin to deteriorate, and could in an extreme case result in adding noise and slight instability by injecting random predictions into a current local region data. To prevent from occurring, the comparator 250 output turns off the predictor, and disables any adjustment to a current local region data, by forcing the multiplexer 270 output to zero. The encoder then uses the original unmodified current local region data instead, as disclosed in existing patents. The encoder 290 may use any appropriate encoding, such as compression algorithms (either lossless or lossy) including PM or PCR compression as used with any other compression technique, e.g., wavelet compression, LZW compression, DCT compression, Huffman encoding, etc. One example of a form of compression is described in U.S. Patent No.9,385,751, herein incorporated by reference in its entirety.

[0120] As mentioned, the buffer may be a circular buffer. The circular buffer may be initialized to zero data at the beginning of the procedure. This may ensure that the computed correlation level 230 will be below the threshold 240, and the encoder will use the unmodified first few local regions data, until previous local region data becomes available (since for the first few current local regions no previous local regions data is yet available).

[0121] FIGS. 2-5 illustrates one example of a process of determining the level of correlation between a current local region series data and a previous local region data, and the use of a predictor to substantially reduce the amount of information in the encoded stream, in order to provide a significant improvement in compression, as disclosed herein. Although mass spectrometry data is used to illustrate the method in these examples, it should be understood that these techniques and apparatuses are not limited to these methods. Indeed, these methods may be used for other multidimensional ordered series data sets, as described herein.

[0122] In general, mass spectroscopy typically involves hitting a sample with high energy laser beam to release a “cloud” of charged particles which are directed by an electric field to a detector, and the instrument measures the time-of-flight of charged particles. Depending on the202515949laser energy, the material is fragmented into many charged pieces. The smaller more agile fragments reach the detector first, faster than the heavier fragments (but you get to see everything, all possible fragments, hence the name: spectrum). The detector measures the intensity (how many fragments of each type), and the time-of-flight (the time it took each fragment to reach the detector). The time-of-flight is proportional to the fragment mass over charge ratio (because heavier higher mass fragments take longer to reach the detector; and the force driving the fragments is directly proportional to the charge, a fragment which is doubly charged will move twice as fast, and reach the detector in half the time). Therefore, mass spectroscopy data generally includes a set of indexed events, for each event, a pair of (x,y) measurements are taken, where x is the observed m / z (mass over charge), and y is the observed intensity (count of how many same fragments detected, as measured by the total current). The set of events includes all the various fragment pieces. Mass spectroscopy data typically includes many peaks, large and small, with many empty gaps in between the peaks, where no fragments were detected. The data is sparse in the sense that there are a few tall peaks, interspersed with many more small peaks and gaps. The observed y intensity values can have a large dynamic range: from a single count and up to ten orders of magnitude (that is, 10,000,000,000, or 10 billion count). The observed x values are measured in units of Daltons, and are measured at very high accuracy, of about 1 part-per-million or better. A fragment that weighs 10,000 Daltons, at 1 parts-per-million, will also require a ten order of magnitude range. The only way to represent numbers with such a wide dynamic range is by using double-precision floating point representation. The common range of integer numbers used by computers is simply insufficient. In addition, to improve the detection capability, sometimes the fragments are forced to go through a tube filled with inert gas. The fragments collide with the gas molecules, increasing the time-of-flight. The small fragments have no trouble squeezing between the gas particles, but the larger ones keep colliding with gas molecules on the way, getting slowed down. This method is called ion mobility. By measuring how the heavier particles are slowed down by the gas, this provides additional information that can be used to distinguish between ion fragments of similar weight, that would otherwise be reaching the detector at the same time without the gas. The ion mobility value may be a third parameter z in this case, being measured for each event.

[0123] FIG. 2 shows just one example of a ‘typical’ current local region data of mass spectrometry data, showing a set of indexed series data 403 and 404, in which each indexed data202515949is comprised of an index (n), a first variable (xn) which in this example is a mass-to-charge ratio, m / z, and a second variable (yn), which in this example is the intensity abundance count of ion fragments with the given m / z. The horizontal axis in FIG. 2 is a mass-to-charge ratio, and the vertical axis is the intensity ion fragment count.

[0124] As shown in FIG. 2, a plot 420 shows a typical ordered series data, recorded by a mass spectrometer. The recorded data represents a portion of one spectrum, of the relative intensity abundance of charged ions as a function of the mass to charge ratio (m / z). The horizontal axis in this figure is the mass-to-charge ratio (m / z), and the vertical axis shows the observed intensity values, or relative ion abundance. Other examples of ordered series data that may be compressed as described herein may include (but is not limited to) microscopy data, multiplexed fluorescence microscopy data, and any data where the horizontal axis is a time axis, or a wavelength or frequency axis, as the case may be.

[0125] In a typical mass spectrometry experiment, a sample (which may be solid, liquid, or gas), for instance a piece of cell tissue or a drop of blood, is vaporized by a stream of electrons or other means, such that the sample is ionized, and an electromagnetic field is applied to accelerate the charged ions and separate them according to their mass-to-charge ratio. Ions of the same mass-to-charge ratio will experience the same amount of deflection. Finally, the ions are detected by a detector that is capable of detecting charged particles. The results are usually displayed as spectra of the relative abundance of detected ions as a function of the mass-to-charge ratio. The molecular composition of the sample can then be identified from the characteristic fragmentation pattern, combined with the known elemental masses.

[0126] In each experiment, a mass spectrometer can produce many such spectra over time, resulting in a large quantity of data. A typical output in profile mode can readily exceed tens or even hundreds of GB of data from one single experiment (1 GB = 1024 x 1024 Kbytes). A mass spectrometer can be used in genomics and proteomics for sequencing genes and amino acid peptide chains, and for determining the molecular composition of chemical compounds, as well as for identifying abnormal disease patterns such as altered glycosylation in cancer.

[0127] Each mass spectrometer scan is comprised of ordered series data, the data comprised of a series of peaks 403 and 404 of varying intensities at particular m / z locations. Each of the sequence of peaks in 420 is comprised of an (x, y) pair of double precision floating point numbers: the y-value is the observed intensity abundance, or the amplitude of a particular peak,202515949and the x-value is the observed mass-to-charge ratio m / z. For example, the intensity abundance of the second peak is about 160.0, observed at a mass-to-charge ratio of about 341.2 m / z, as represented by the pair (341.2, 160.0) of double precision floating point numbers. The charged ions are trapped in an ion trap and released at particular times, at which point they travel under the influence of an electromagnetic field towards the detector, to be observed as a series of m / z and ion abundance values over time. Except for metadata and the experiment setup information, nearly all the data recorded is comprised of sequences of indexed paired double precision floating point numbers.

[0128] Each scan of the mass spectrometry data is comprised of a variable number of (x, y) double precision peaks, the instrument is usually programmed to record peaks in a given range of mass-to-charge values, between a given minimum and maximum m / z values of interest, for example between m / z=100 to m / z=5000 Dalton. The minimum and maximum m / z values usually remain static for all scans belonging to the same level spectra. Any peaks outside of the given mass-to-charge range, with m / z values lower than the minimum m / z value, or larger than the maximum m / z value, are discarded.

[0129] The entire scan range (for example, between m / z=100 to m / z=5000 Dalton) can then be divided into N overlapping or non-overlapping local regions, where each local region, representing a portion of a scan, is treated sequentially, one at a time. For example, the entire range of 100-5000 Daltons can be divided into seven non-overlapping local regions, having a range of 700 Daltons each. As mentioned above, the number (N) and / or sizes of these different overlapping or non-overlapping regions may be set in any appropriate manner, and may be based, e.g., on the size of the range, or a predetermine, user-defined and / or automatically or semi-automatically adjusting number and / or size. The plot 420 in FIG. 2 shows an example of data from a first local region of a given scan, with a range of mass-to-charge ratio between 100 to 800 Daltons.

[0130] In this example, the entire mass-to-charge ratio range of 100 to 5000 Dalton is comprised of 7 non-overlapping local regions, with a range span of 700 Dalton mass-to-charge ratio each. In practice, however, it may be desirable to have some overlap between subsequent local regions, whereby the span of each local region is extended by a certain amount on either side, for example by 100-150 Daltons on either side, anywhere from 5% up to 50% overlap. An overlap may provide smooth transitions between local regions without creating artifacts along202515949local region borders. In addition, the overlap may be useful for eliminating limit cases, such as when a significant high-intensity peak happens to fall exactly on a local region boundary, where even a small rounding error may shift the position across to the other side of a local region border, hence it may not always be consistently included in a current region, which could introduce some instability into the prediction. Therefore it is often preferable to include some overlap, by extending the local region span on either side, in order to prevent instability and minor artifacts.

[0131] The plot 410 in FIG. 3 shows an example of a first local region data, with a range of mass-to-charge ratio of 100 to 800 Daltons, of a corresponding previous scan data, captured by the instrument at the most recent time before the current scan data. In FIG. 3, showing an example of a corresponding previous local region data, including a set of indexed series data 401 and 402, each indexed data comprises an index (ri), a first variable (xn) (mass-to-charge ratio m / z) and a second variable (yn) (intensity abundance count of ion fragments with the given m / z). The horizontal axis is mass-to-charge ratio, and vertical axis is the intensity ion fragment count. Since the current scan and the previous scan are taken in close time proximity, the ion content is likely to be similar to a certain extent, and likely to have at least some similar peaks in common. The plot 410 data comprised of a series of peaks 401 and 402 of varying intensities at particular m / z locations.

[0132] Comparing the peaks FIGS. 2 and 3, one can readily recognize the series of 404 peaks in current local region of FIG. 2 being similar to the series of peaks 402 in previous local region FIG. 3. The peaks appear to have almost the same m / z location (within the instrument tolerance level, which can be highly accurate to 1 part-per-million in certain instruments), and similar intensities pattern, the intensity is typically less accurate than m / z. In addition, there are many peaks 403 in current local region 420, and other peaks 401 in previous local region 410, which clearly do not align. The probability of uncorrelated peaks falling by chance within 1 parts-per-million mass-to-charge ratio of each other is extremely small.

[0133] A further confirmation of correspondence between the series of peaks 402 and 404 may be obtained by the spacing and number of peaks in each peak series. For an isotope series, the spacing between subsequent peaks is almost constant, indicating the charge state of a molecule. The number of peaks and the intensity distribution of isotope peaks generally follows an averagine model, which is based on the elemental composition of the observed molecule.202515949Hence the similarity of the series of the peaks 402 and 404, having the same overall mass, spacing, and intensity distribution pattern provides a strong indication of the same molecule being observed in both scans.

[0134] The level of correlation may be computed from the series of similar peaks that exist between the current and previous local regions. For example, peaks 402 and 404 are considered similar if the majority of peaks have one or more of: (1) almost the same mass-to-charge ratio m / z, within the instrument accuracy limits, (2) the same charge state as determined from nearly equal spacing between subsequent peaks, and (3) similar peak intensity abundance distributions that match the averagine model. In addition, each of the peaks under consideration must be of a significant intensity, sufficiently above the instrument signal-to-noise floor, to prevent false-triggering on random low-intensity noise peaks. The combination of these factors, together with a distinct isotope-resolved pattern, may provide a robust and highly reliable measure of the correlation level. In other systems, correlation may be based on similar features, such as intensity, wavelength, relative spacing / position, etc.

[0135] The resulting level of correlation between a current and one or more previous local regions may then be compared to a threshold. The threshold may be predetermined or calculated. If the level of correlation is larger than or equal the threshold, a predictor is used for adjusting the current local region data, based on the previous local region data, to reduce the amount of information, before encoding the current data. Otherwise, if the level of correlation is smaller than the threshold, no prediction is used, and the current local region data is directly encoded without adjustment, since low correlation levels may be indicative of weakly related peaks, which may not be as reliably predicted, and could potentially result in adding more noise and reducing the compression. The correlation level threshold is carefully chosen to ensure significant compression improvement when exceeding the threshold.

[0136] A predictor may be used to predict the current local region data, when the level of correlation meets or exceeds the threshold, from one or more corresponding previous local region data. The use of more than one corresponding previous local region data can improve the peak signal to noise ratio, by averaging the data, reduce fluctuations in peak intensities, and generally improve the prediction quality. However, this may come at the expense of requiring more memory to store multiple previous local region data of additional scans.

[0137] FIG. 4 illustrates one example of the method of predicting the correlated portion of a202515949current local region from the previous local region data, shown in plot 430.

[0138] In FIG. 4, the illustration shows a set of correlated data between a current local region data 410 (e.g., as shown in FIG. 3) and previous local region data 420 (shown in FIG. 2). Only some of the data, such as the peaks 404 in FIG. 2, correlate with the peaks 402 in FIG. 3. The other uncorrelated peaks 403 and 401 in FIGS. 2 and 3 have been discarded. Correlated peaks must have certain properties in common, for example, in the MS data shown in FIGS. 2-4, correlated peaks show: (1) almost the same mass-to-charge ratio m / z value, within the instrument accuracy; (2) almost equal spacing between subsequent peaks, confirming a certain charge state; and (3) the observed intensity distribution should generally follow the averagine model. Note the data 402 is drawn slightly shifted away from 404 only for illustration purpose, in practice the peaks 402 will fall almost on top of 404, within the instrument m / z accuracy tolerance of 1 parts-per-million typical.

[0139] As shown in plot 430 in FIG. 4, only the peaks with high level of correlation 402 and 404 are retained from the previous and current local region data, respectively. All uncorrelated peaks 401 and 403 from either the previous FIG. 3 or current local region data FIG. 2 have been discarded. The predictor calculates the optimum scale factor to bring the correlated previous local region peaks as close as possible to match, i.e., predict, the set of correlated current local region peaks. Note that since individual peak intensities are not as accurate and generally deviate to some extent between scans, it is generally not possible to perfectly align all the highly-correlated peaks together.

[0140] In some variations the optimum scale factor may be chosen to: (1) match at least one of the highly-correlated peak intensities, while simultaneously (2) minimizing the sum of differences between all the other highly-correlated peak intensities, and (3) ensuring that each individual highly-correlated peak intensity difference must be greater or equal than zero. In some variations, the scale factor may be chosen to minimize the sum of differences between all the other highly- correlated peak intensities. In some variations the scale factor may be chosen to both minimize the sum of differences between all the other highly- correlated peak intensities and to ensure that each individual highly- correlated peak intensity difference must be greater or equal than zero.

[0141] The prediction according to one example is illustrated in FIG. 4 plot 430. As mentioned, only the highly-correlated peaks are shown. The current local region correlated peaks202515949404 are shown unsealed, in solid lines, exactly same as in FIG. 2. The predictor scales the previous highly-correlated local region peaks 402 (FIG. 3), shown in dotted lines, to match the current 404 peaks as close as possible. The optimum scale factor in this example is when the third 402 peak from left is scaled to the same height as the corresponding current third 404 peak intensity. The other 402 peaks, first, second and fourth from left are scaled by the same amount as the third peak, and in general are not likely to exactly match their corresponding 404 peak intensity. Note the 404 and 402 peaks in plot 430 are deliberately drawn side-by-side for illustration purpose, in practice they are likely to be extremely close to each other, within the instrument accuracy tolerance of about 1 parts-per-million.

[0142] It can be appreciated from plot 430 that the optimum threshold can be obtained by matching the intensities of the third peak from left in this example. Since all the other 402 intensities (e.g., the first, second and fourth peak from left) are lower than their corresponding 404 peak intensities, for each of the other scaled 402 peaks. This ensures that the individual difference between a 404 peak and corresponding scaled 402 peak is always greater than or equal than zero according to condition (3). If the procedure and / or apparatuses matches the intensity of any peak other than the third from left in this example, the third 402 peak would be scaled to become taller than its corresponding 404 peak, and at least the difference between the third 404 peak and corresponding scaled 402 peak would become negative, thereby voiding condition (3) in this method. Condition (3) is useful for ensuring that the prediction residue, after subtraction of the predicted intensity, will remain positive, since mass spec intensity data represents a positive ion-abundance count, which cannot typically be negative.

[0143] In another variation, in which the data is microscopy data using floating point representation, where data is allowed to be either positive or negative, a tighter optimum scale factor can be found by eliminating condition (1) and (3) above, and allowing the differences between individual 404 and corresponding scaled 402 peak intensities to assume either positive or negative values.

[0144] A predictor may thus utilize a single scale factor to predict a subset of highly correlated peaks in the current local region 404 based on the intensities of the correlated peaks 402 in a previous local region. The optimum scale factor may be encoded into the compressed stream, a single value for each local region, for use during decompression. The local region data is then adjusted by subtracting the predicted correlated peaks 402 from the current local data202515949before encoding. An example of this result is shown in FIG. 5 plot 440.

[0145] FIG. 5 shows an example of an adjusted current local region data, after subtraction of predicted correlated data 430 using the data shown in FIG. 4. The adjusted data comprised of residues 405, after subtraction of predicted correlated previous peaks 402 from original current local region peaks 404 as shown in FIG 4. The uncorrelated current local region peaks 403 are not adjusted. The result of the adjustment is the replacement of high-intensity peaks 404 with much smaller residues 405, as highlighted by dotted line 450, and as can be seen by comparing FIGS. 2 and 5. The adjusted data may then be encoded into the compressed stream using any appropriate compression technique. See, e.g., U.S. 9,385,751, describing “PMI compression”.

[0146] The local region data after subtraction may comprise the uncorrelated local region peaks 403 (FIG. 2) and residual intensities 405 of the correlated peaks remaining after subtracting the predicted intensities based on the previous local region data. The difference becomes evident by comparing the original local region data in plot 420 (FIG. 2) with the adjusted local region data in FIG. 5 plot 440. The group of tall high-intensity peaks 404 in FIG. 2 have been replaced by a considerably smaller group of intensity residues 405, as highlighted by the ellipse 450. The difference in intensity magnitude can be huge, up to several orders of magnitude in a typical mass spectrometry data. The resulting magnitude of the residues 405 intensities depends on the prediction accuracy, which can be quite good for a molecule being repeatedly scanned on the instrument, as is quite often the case with this type of data.

[0147] In this example, the prediction residues 405 are being encoded into the compressed stream, instead of the original high-intensity peaks 404. When the prediction is effective, the resulting intensities of the 405 peaks are very small, which leads to a significant improvement in compression.

[0148] During decompression, the previous local region data has already been restored before the current region data is being processed. An identical predictor may than used to scale the correlated peaks of the previous local region by the decoded scale factor, and add them back to the current local region data. The adjusted current local region data is then decompressed, to restore the adjusted current local region data comprised of 403 peaks and 405 residues. The scaled predicted peaks of the previous local region may then be added to the adjusted current local region data, to restore the original local region data 420 (e.g., as shown in FIG. 2).

[0149] If the level of correlation is weak, as indicated by being smaller than the given202515949threshold, no prediction is used, and the local region data is encoded as is, without adjusting the data.

[0150] FIG. 6 shows a three-dimensional elution profile of a typical mass spectrometry data over time. Each ion fragment exhibits a typical bell-shaped curve. The intensity is low at the beginning, then gradually rises to reach an apex, then gradually goes down to zero. The time duration and apex intensity height can vary based on the fragment composition. There may be some overlap between co-eluting ion fragments on rout to the detector.

[0151] Referring to FIG. 6, plot 510 illustrates a typical elution profile of ion fragments in a mass spectrometry instrument with liquid chromatography. The fragmented ions are held in an ion-trap and released in a certain order toward the detector according to their mass-to-charge ratio m / z. For example, ion fragments 501 are first to be released in time, followed by ion fragments 502, then 503, and finally 504 are last to be released in plot 510.

[0152] A typical elution profile of each ion fragment is an approximate Gaussian curve. The observed intensity is low at the beginning, then gradually increases as more molecules are being fragmented, reaching an apex at the center of the elution profile, and then dropping back down. The time duration and apex intensity of each molecule 501 through 504 can vary based on the composition of each molecule. The combined signal, as observed by the detector at any given point in time is shown in FIG. 7 plot 520.

[0153] FIG. 7 shows an observed two-dimensional elution profile of a typical mass spectrometry data such as that shown in FIG. 6, after integration over the mass-to-charge ratio m / z axis, over time, also known as Total Ion Current (TIC). The total ion current is proportional to the elution profile envelope, the maximum of the individual curves 501-504 at any given time. Since ions are electrically charged, a direct measurement of the detector current is typically proportional to the elution envelope. The minima points of the elution envelope define the crossover transition time-points t2, t3, t4 from one ion fragment species to another.

[0154] As can be appreciated from FIG. 7 plot 520, one type of ion fragment is dominant for most of the time. The ion fragment 501 is dominant from tl to t2. Initially, there is a pure concentration of only ion fragments 501, however, toward the end time t2 the amount of ion fragments 502 gradually increases until at time t2 it reaches an equal amount to ion fragments 501, wherein 501 no longer dominates. Ion fragments 502 dominate from time t2 to t3, and so on. Each ion group strongly dominates near the apex of the ion-group, and becoming more of a202515949mixture spectra closer to the transition points, at times tl, t2, t3 and t4.

[0155] A typical mass spectrometer run takes one to two hours, while the typical elution duration of any particular ion fragment 501 to 504 may last from a few minutes up to 25 minutes or so, depending on the experiment and the particular material composition of the molecules of interest. Increasingly, mass spectrometry instruments can capture scans at high speed, higher than 100 scans per second without losing sensitivity or resolution. This means that subsequent scans are likely to be very similar throughout an instrument run for the most part, except for a few small intervals, during which the eluting ion-fragments transition from one type to another. The proposed methods and apparatuses described herein may successfully predict the correlated current local region data from the previous data with high accuracy, and achieve much higher compression than would otherwise be possible by compressing each scan independently. Thus, instead of spending many bits to encode the similar content of each subsequent scan over and over again, as would be the case with independent compression of each scan, the correlated portion may be compressed only once at the beginning, and in subsequent scans, only the residual changes, if any, together with uncorrelated peaks, are encoded into the compressed stream, leading to a significant increase in compression according to the methods and apparatuses described herein.

[0156] In some cases, mass spectrometry data may include mixed data. For example, as described above, the data may include a mixture spectra from overlapping or partially overlapping isotope intensity distributions or multi-sample elution profiles. As such, in some cases, the methods described herein for compressing mixed data may be applied to mass spectrometry data.

[0157] As discussed, the methods described herein may be used to compress image data, such as histopathology image data. Histopathology image data may be sparse in that it include relatively few pixels with high intensity change and many pixels with lower-intensity changes. FIGS. 8A-8B illustrate one example of a side-by-side comparison of imaging (histopathology) data from un-compressed data (FIG. 8A) as compared to data compressed as described herein (FIG. 8B). In this example, a small portion of the image is shown at a magnification of 40x as compared to the originally captured image, to show the high fidelity of this technique. The two images are visually indistinguishable, even at this high magnification level. The original (FIG.8A) file size is approximately 1764 MB (megabytes), and was compressed as described herein,202515949to a compressed file size of, e.g., 15.9 MB (compression ratio of 111 : 1). For example, the original data may be compressed as described herein by first dividing the imaging dataset (which is a multidimensional ordered series data) into a plurality of local regions. In this example, the local regions may be sub-regions (e.g., square or rectangular sub-regions, though any two-dimensional shape may be used). These regions may be overlapping or non-overlapping. The imaging (e.g., pixel) data in each local region comprises at least one indexed data set, comprising an index (in this case, a spatial index, such as x, y index) may be used) within a given local region of the image and one or more variables that are indexed by the index (such as the pixel intensity, brightness, color, etc.).

[0158] Each local region may be processed in an order (e.g., a raster order, such as left to right, top to bottom, etc. or any other consistent order), to process a current local region, one local region at a time, as the data set is processed. As described herein, for each local region, the method / apparatus may identify a current local region, then identify and compare this current local region to a previous current local region (the initial or first few starting current local region(s) may be left unadjusted). For example, the previous current local region may be a region that corresponds an offset region that has already been processed, e.g., above and / or to the left of the current local region in an image dataset, when processing from left to right, top-to-bottom. If processing in a different order or pattern, the previous local region may be chosen from a different portion that has already been processed. In some variations more than one previous local region may be compared, or in a different order. For example, in some variations two or more previous local regions may be combined (e.g., averaged) or the like to improve the signal to noise ratio. The current local region may be compared to (one or more, or a combination of) previous local region(s) and a correlation level may be determined (e.g., computed) for this comparison. For example, the corresponding index region (e.g., x, y values) between different, and in some case partially overlapping) regions may be compared and a value (e.g., a correlation level) may be determined. The correlation level may be determined by comparing the intensity of each corresponding region (e.g., index, such as x, y positions) within the local regions. Where the values are highly correlated for the index values, in one or more variables, the correlation level may be increased by a set amount.

[0159] This computed correlation level may then be used to further process the data of the current local region. If, for example, this comparison / correlation level or value exceeds a202515949threshold, which may be preset or adjustable, then a predictor may be identified and used to modify the current local region. A predictor may calculate each of the one or more variables as a function of the index, between a current local region data, and at least one previous local region data. For example, the predictor may be a scaled version of those regions (index values) of the current local region that have a sufficiently high level of correlation in a previous local region. The value at each index may be 0 (e.g., for regions that are not corresponding) or some scaled value of the corresponding indexed value from the previous local region(s). The scaled value may be scaled based on the entire local region, and may be scaled to a value that corresponds to a closest predicted value of one of the indexed peaks (e.g., pixel intensity, etc., in an image data set).

[0160] The current local region may be adjusted using this scaled predictor. For example, the current local region may be processed by adjusting the current local region data to subtract the scaled predicted correlated data based on the previous local region data when the level of correlation exceeds a threshold (e.g., between the current local region data and the previous local region data).

[0161] The modified (e.g., adjusted) current local region may then be encoded by an encoder, such as a PMI or PCR encoder, or by using standard ZIP or other compression encoder. The current local region data may be encoded along with the scaling factor and / or other data related to the predictor, into the compression stream.

[0162] The image may be uncompressed and restored (as shown in FIG. 8B) by decoding the compressed data, identifying (from the processed / compressed data) the local region(s) to be restored, identifying (from the processed / compressed data) the predictor used to modify the portion(s) of the data set, and adding back the scaled predictor to the local region to be restored to form the restored local region. FIG. 9 schematically illustrates one example of a method (and apparatus) for restoring all or a portion of a data set compressed as described herein.

[0163] In FIG. 9, the compressed data 905 may be decompressed by decoding the adjusted local region information 910 from the compressed input data, as well as the predictor data (e.g., scale factor 975 or predictor scale factor), from the compressed data. Both the adjusted local region to restore and the predictor data may be decoded using a decoder 907 that corresponds to the type of encoding used to encode the data (see FIG. 1). Thus, the decoder may provide a decoded current local region data 910 to be restored, and the decoded predictor scale factor 975202515949corresponding to this target local region to be restored. By reversing the procedure described for compressing in FIG. 1, the predictor and the decoded local region may then be added together 980 to provide the restored target (“restored local”) region data 990, which may be output, displayed, saved, etc. This process may be iteratively used to decompress and restore an entire file, and / or it may be used just to restore a portion of a dataset.

[0164] For each local region in turn, the compressed data is decoded and applied to restore the adjusted local region data shown in FIG. 5. The optimum scale factor 975 is also retrieved from the compressed data. Each local region may be processed in an order (e.g., a raster order, such as left to right, top to bottom, etc. or any other consistent order), to process a current local region, one local region at a time, as the data set is processed. As described herein, for each local region, the method / apparatus may identify a current local region, then identify and compare this current local region to a previous (already processed) local region. For example, the previous local region may be a region that corresponds to an offset region that has already been processed, e.g., above and / or to the left of the current local region in an image dataset, when processing from left to right, top-to-bottom. If processing in a different order or pattern, the previous local region may be chosen from a different portion that has already been processed. In some variations more than one previous local region may be compared, or in a different order. For example, in some variations two or more previous local regions may be combined (e.g., averaged) or the like to improve the signal to noise ratio.

[0165] The current local region 910 may be compared to (one or more, or a combination of) previous local region(s) 920 to identify a subset of peaks which may have been adjusted as disclosed herein. The Identify & Predict module 960 searches the content of the local and previous region data, looking for a sequence of relatively high intensity candidate peaks in the previous region data (peaks 402 in FIG. 3) that match the residue peak m / z locations (peaks 405 in FIG. 5). The identification can be made with high confidence by taking advantage of the high m / z accuracy of the mass spec instrument, typically accurate down to 1 part-per-million tolerance. The identified peaks must match the residue locations, and the uniform spacing between subsequent peaks must match the corresponding charge state of the peaks in the previous local region data. In addition, the intensities of the peaks in the previous local region must match the expected averagine distribution as already disclosed herein.

[0166] Once an identification is made by the module 960, of a candidate sequence of peaks202515949in the previous local region that matches the m / z residue locations in current adjusted region data, an identical predictor (to the one used during encoding) is applied for predicting the local region peaks based on the identified peaks in the previous local region data. Each of the identified peaks in the previous local region is scaled by the optimum scale factor, extracted and decoded from the compressed stream data. The scaled peaks 965 are shown in FIG. 9.

[0167] The predicted scaled sequence of peaks 965 is added to the adjusted local region data 910 through the multiplexer 970 and the summing node 980. By adding the predicted scaled peaks (predicted local region peaks, 402 in FIG. 4) to the adjusted local residues (adjusted local region data, peaks 405 in FIG. 5), the original local data (peaks 404 in FIG. 2) is effectively restored to the original local region values. The restored local region data is now available at the summing node output 990, and provided to the correlation level module 930 in order to verify the restored local region data is indeed correct and accurate.

[0168] The restored current local region data 990 may be compared to (one or more, or a combination of) previous local region(s) and a correlation level may be determined (e.g., computed) for this comparison. For example, the corresponding index region (e.g., x, y values) between different, and in some case partially overlapping) regions may be compared and a value (e.g., a correlation level) may be determined. The correlation level may be determined by comparing the intensity of each corresponding region (e.g., index, such as x, y positions) within the local regions. Where the values are highly correlated for the index values, in one or more variables, the correlation level may be increased by a set amount.

[0169] The computed correlation level may then be used to validate the data of the current local region. If, for example, this comparison / correlation level or value exceeds a threshold 940, which may be preset or adjustable, then a multiplexer 970 may be turned on to retain the current local region data. However, if the level of correlation between the current local region data and previous local region data is below the threshold 940, the multiplexer 970 is turned off, thereby disabling the scaled peak corrections from being added, and merely passing the adjusted local region data 910 uncorrected to the output 990. Hence the restored local region data at the decoder output 990 accurately reflects the original input to the encoder (by using the adjusted local region data based on the predicted peaks when the level of correlation exceeds the set threshold, or using the unadjusted local data when the level of correlation is below the threshold).202515949

[0170] The local region data 990 is thus restored with high fidelity and made available for viewing or storing or for any subsequent downstream processing.

[0171] For example, FIG. 10 illustrates how the compression methods and apparatuses described herein may be used to provide direct (e.g., random) access from a compressed data stream. For example, in FIG. 10, a large data set 1002 (e.g., an image data, such as the large image shown pre-compression) may be compressed and mapped into a data stream 1004 (also referred to as a compressed bit stream). Just a portion of this large data set 1002 may be analyzed or manipulated. For example, a region 1006 may be extracted without having to decompress the entire data set 1002, allowing processing of only a particular one or more target region(s). More specifically, only bit subset of bits 1008 within the data stream 1004 need be accessed and decompressed to retrieve region 1006. This is in contrast to other compression techniques that require decompressing the entire dataset (e.g., an entire image) to have access to a particular region. This offers some immediate technical benefits over prior solutions: only the local regions that comprise the portion of image of interest need to be restored and processed, the rest of the image is not decompressed.

[0172] FIGS. 11 A-l IB illustrate another example of a dataset that may be compressed as described herein. In this example the data set is a multiplexed florescence dataset, showing five color staining (of five different biomarker dyes). The entire sparse multidimensional ordered series dataset may be compressed as described herein and regions may be decoded and examined, including magnifying de-compressed regions without significant loss of detail, from a zoom of 15% to 1600%. For example, in FIG. 11 A, the boxed region B may be decompressed and magnified, as shown in FIG. 1 IB. FIG. 1 IB shows a magnification of 15% zoom. The original file size of the image in FIG. 1 IB is 321 MB; the methods described herein may be used to compress this dataset with a compression ratio of at least 39: 1, providing a compressed file size of approximately 8.2 MB. The processed and uncompressed image region is indistinguishable from the original region. For example, the per pixel mean square error (PMSE) in this example is 4.24e-4, and the peak signal-to-noise ratio (PSNR) is approximately 56.25 dB. This compares very favorably to other commonly used compression techniques, such as JPEG (having a typical PSNR of about 45 dB at Q=90). On average, other image types had similar or better compression ratios at comparable quality; for example, data sets of images over 1 GB in size were compressed using the methods and apparatuses described herein to greater than 150:1202515949(e.g., from 1,628 MB to less than 14.4 GB, providing a ratio of about 154:1).

[0173] FIG. 12A illustrates another example of a large imaging dataset that may be advantageously compressed as described herein. The original image shown in FIG. 12A was compressed using a standard JPEG image compression technique at high compression and using the methods described herein (referred to PMI compression or PCR compression). Under magnification, the methods described herein introduced substantially less artifacts and loss than other known compression techniques, including JPEG. For example, a small region of the image of FIG. 12A was examined following compression and restoration using the methods described herein (shown in FIG. 12B) and are compared to the same region following compression and restoration using JPEG, shown in FIG. 12C. As can be seen by this comparison, the JPEG compression of FIG. 12C introduced a significant artifact not present when processed as described herein and shown in FIG. 12B.

[0174] As discussed, in some examples, the methods may be adapted to compressing mixed data, such as flow cytometry data, which is typically mixed due to overlapping emissions from multiple dyes or fluorophores within narrow-wavelength detection bands.

[0175] Unlike existing compression techniques, the methods described herein may achieve order(s) of magnitude higher compression than is currently possible with conventional compression methods such as GZIP, LZW, 7ZIP, BZIP2, LZ4, ZLIB, or ZSTD. The method is best suited for compressing double or single precision floating point data precisely of the type recorded by a variety of sensors and analyzers, such as flow cytometry spectrographic data. In stark contrast with existing compression techniques, which make no assumptions about the data, the present method may achieve far higher compression ratio by taking advantage of the data structure and the fact that the data, for the most part, tends to be sparse in a particular narrow sense, as described herein. This is not to say that there are a few data points overall.

[0176] Instead of treating the data as ASCII text strings, which it isn’t, and looking for longest randomly occurring patterns of strings, which are hardly expected to be consistent or even predictable, like existing compression methods, the methods described herein are designed to identify the likely aspects of data that are bound to be consistent and predictable, and proceed to compress each data segment independently, using personalized code words, in the most effective manner possible. Thus, the methods and apparatuses described herein are specifically202515949tailored to the compression of the sparse multidimensional ordered parallel series data in double precision floating point format, such as in particular, flow cytometry spectrographic data.

[0177] FIG. 13 illustrates examples of flow cytometry principles. A high-power laser light source LS (1301) emits a narrow light beam (1302), aimed at a transparent channel T1 (1303), fixed at a plane perpendicular to the light beam. The cells or particles to be analyzed (1304) are suspended in liquid, stained with dyes and treated with specific antibodies and binding reagents to bind various different fluorophores to desired chemical receptors and biological features.

[0178] The liquid with the treated cells or particles flowing through a narrow channel T1 (1303), in a direction from top to bottom, as indicated by the flow arrow. Using this method, the cells or particles (1304) are made to pass through the light beam (1302), ideally one at a time. The channel may be placed at a sight angle at the perpendicular plane, the top positioned slightly further away while the bottom positioned slightly closer to the viewer.

[0179] When a cell or particle (1304) crosses the light beam, part of the light is absorbed by the cell or particle, and part of the light is reflected and scattered in various directions. The main light beam (1302) travels through a first semi-transparent mirror (1305), placed at 45 degrees to the light beam. Most of the light is reflected by the mirror (1305), and directed through a series of similar subsequent semi-transparent mirrors. A portion of the light, however, goes through the mirror, since the first mirror (1305) is semi-transparent, and reaches a first forward direction FD (1306) detector.

[0180] The first detector FD (1306) monitors the light beam intensity in the forward direction. When a cell or particle (1304) passes through the light beam, the intensity observed by the first FD detector drops, because the cell or particle partially blocks some of the light. The intensity observed by FD detector (1306) is reduced because the cell or particle absorbs some of the light, and at the same time also reflects and scatters some light in other directions aside from the main beam direction. On the other hand, when a cell or particle is not present, the light beam travels unimpeded, at maximum intensity, a portion of the maximum intensity reaches the first FD detector (1306).

[0181] A sudden drop in the observed FD intensity signals the arrival of a cell or particle in the light beam path. As the cell moves into the light, the intensity becomes lower, until the cell or particle is fully illuminated by the beam. Eventually, the light intensity starts to increase again when the cell or particle begins to exit the light beam path on the other side. The lowest intensity202515949point is used as a trigger to tell when a cell or particle is fully present in the light beam. Thus the first FD detector (1306) is used as a detector to identify the presence of a cell or particle in the light beam, and trigger all other detectors to take measurements at the same time.

[0182] The light beam reflected by the first semi-transparent mirror is routed through a cascaded series of subsequent semi-transparent mirrors (1305), each subsequent semi-transparent mirror diverts part of the light to a corresponding narrow- wavelength band-pass filter Wl, W2, W3 (1309), each band pass filter is tuned to pass a different narrow portion of the light spectrum centered around a specific wavelength. Only light within a narrow range around the particular wavelength is allowed to pass through the narrow-wavelength filter (1309). All other wavelengths of the incoming light are blocked by the narrow- wavelength filter, and cannot pass through.

[0183] When a cell or particle (1304) is stained with dyes or otherwise treated with specific antibodies and binding reagents to bind various different fluorophores, each particular dye or fluorophore is excited by absorbing light at one wavelength range, and emitting light at a different, usually higher wavelength. There are many standard dyes and fluorescent reagents available. The set of narrow- wavelength filters (1309) are selected to match the expected emissions from corresponding sets of dyes and fluorescent reagents applied for treating the cells or particles (1304). Thus each narrow-wavelength filter (1309), Wl, W2, W3 is tuned to a specific wavelength of a dye or reagent emission response.

[0184] A set of parallel intensity detectors DI, D2, D3 (1310) are set to measure the incoming light intensity at the outputs of each tuned narrow- wavelength filter Wl, W2, W3 (1309), respectively. There can be many narrow- wavelength filter-detectors pairs in a particular machine, up to 100 or more, the exact number depends on the experiment and the number of dyes and reagents being used. All detectors are triggered to measure the intensity at the same time, at the moment a cell or particle is passing through the light beam, fully illuminated. The trigger signal is provided by first forward detector FD (1306), observing a drop in the intensity, at the lowest intensity point (Event detected).

[0185] The parallel intensity detectors DI, D2, D3 (1310) are effectively measuring the intensity of light at a set of specific wavelengths, the emissions either occurring naturally from the cell or particle, or due to the treatment of cells and particles with specific dyes and reagents.202515949

[0186] In addition to a first forward detector FD (1306) and parallel intensity detectors DI, D2, D3 (1310), a flow cytometer instrument usually includes additional detectors for measuring the portion of scattered light. Because the scattering properties of a cell or particle (1304) provide additional information about the cell or particle, also useful for identifying the type of cell or particle. For example, scattered light can be used to distinguish abnormal red blood cells in sickle cell disease, where the body produces red blood cells with an abnormal “sickle” shape, like the letter C. The misshapen cells scatter light distinctly differently from healthy round red blood cells.

[0187] Two additional scatter detectors SH (1307) and SV (1308) are shown in FIG. 13. The horizontal scatter detector SH (1307) is measuring the intensity of scattered light at a perpendicular direction to the light beam on the horizontal plane. Normally, when a cell or particle is not present in the light beam path, the light beam flows through unimpeded, the horizontal scatter detector only measures the background light, of low intensity values. However, when a cell or particle (1304) crosses the light beam, a portion of the light is scattered in other directions away from the main beam direction, including the horizontal direction. The horizontal scatter detector SH (1307) measures the intensity of scattered light in the horizontal direction. Similarly, the vertical scatter detector SV (1308) measures the intensity of scattered light in the vertical direction, perpendicular to the light beam.

[0188] The horizontal and vertical detectors, SH and SV respectively, like the parallel detectors DI, D2, D3, are triggered by the first forward FD detector to take measurements at the same time. Flow cytometer instrument usually have more than two scatter detectors, at additional angles beyond the horizontal and vertical, as well as in the backward direction, to measure the intensity of light scattered and reflected back from a cell or particle, traveling in the reverse direction, toward the laser source LS (1301).

[0189] In addition, the scatter detectors, including SH and SV, are used for measuring the background light intensity level, when no cells or particles are present in the light beam path. The measurements are used during the data processing, for cancelling out the constant background light, by subtracting the background light level, created by reflections from the transparent channel T1 (1306) walls and imperfections of optical components such as lenses and collimators.202515949

[0190] Upon each detected event - a cell or particle being fully present in the light beam path - a full set of parallel light intensity measurement are taken simultaneously, from many parallel detectors, including the narrow wavelength filter detectors and scatter detectors. There can be many measurements, over 100 in total, all of which are saved as floating point values to file, including the event time. The process repeats again and again for each detected cell or particle event. In the course of a few minutes of time, millions of cells and particles can pass through the light beam, whereby vast amount of spectrographic data, comprised of a wavelength spectrum of light intensity measurements, is gathered, on the entire cell and particle population being measured.

[0191] In addition to measured parameter intensity values from the array of narrow wavelength filter and scatter detectors, flow cytometry data also includes informative parameters, the values of which are not measured in intensity units. One such example is a time parameter, which is almost always present in flow cytometry data. Instead of recording the time of each event, when a cell or particle crosses the light beam, in standard time units, such as the Linux epoc, the time axis is divided into intervals of set duration, for example, every 0.0001 seconds, the duration is a programmable parameter that may vary for each run. The first cell or particle event, crossing the light beam, is arbitrarily set to time equal zero. Each subsequent following event, the integer multiple number of elapsed time intervals from time zero, times the interval duration, is saved as the event time, in IEEE 754 floating point format. The fractional part, within a last interval is ignored. Since the rate of events in a typical flow cytometry is usually faster than the specified time interval, multiple events may have the same time value, and other time values may not be present at all in the data. For example: 0.0, 0.0001, 0.0001, 0.0003, 0.0003, 0.0003, 0.0004, ... , etc. This means that events 1 and 2 occurred during time interval 1, the next 3 events occurred during time interval 3, and so on. No events were recorded during time interval 2 (2*0.0001 time units). In some examples, the method automatically converts the data to the integer numbers, identifies a trend in the data - the integer values monotonously increasing; fits a predictor to the data, and ends up only needing to encode a stream of minor corrections from predicted values in the compressed output stream, resulting in significantly higher compression for this parameter. It achieves the higher compression automatically, without assuming anything about the data content.202515949

[0192] As used herein, a ‘trend’ refers to a statistical trend, such as a pattern found in time series datasets. A trend may be an upward or downward movement or pattern in in part of all of the data (e.g., for part or all of a time series). A trend may be in any type of ordered series datasets, including many types of analytical data sets (e.g., flow cytometry, mass spectroscopy). In the context of a trend, ‘time’ refers to the index by which the series data is ordered; in flow cytometry case, the series data is typically ordered sequentially by an event index - a cell crossing the light-beam event - upon which, a set of m parameter values are recorded. The time of each event is also recorded as one of the parameters. In the case of mass spectrometry, the series data is ordered by a retention time index at which particular pairs of data (peak intensity, mass over charge) are observed.

[0193] A second example of an informative parameter is the event parameter, comprised of a binary value for each index event (n), a value of 1.0 indicates a valid event, while a value of 0.0 indicates an invalid event, which has not been properly recorded, due to an internal issue. Like all parameters, the values are saved in IEEE 754 floating point representation. In a typical run, the majority of events, if not all, are usually valid. The IEEE 754 floating point representation is obviously rather wasteful in this case, using a 64-bit quantity to represent a single bit. In some examples, the method not only automatically reduces the data to a single bit (64x compression) but also run-length encodes the resulting binary vector (another lOOx compression or more on top, when most events are valid).

[0194] An issue with flow cytometry data is the interleaving of detector measurements. Millions of cells or particles are passing through the laser beam at a high throughput rate.Whenever a cell or particle crosses the light beam, an event is detected, which triggers up to a 100 or more light intensity measurements to be taken in parallel. Since the cells are passing at high flow rate, there is no time for processing or re-ordering the data, all the flow cytometer can do is merely write the data out as fast as possible.

[0195] Since the data is written out sequentially, one event after another, in the same repeating measurement order for each event, the data is inherently interleaved. Consider, for example, a flow cytometer capturing 100 different light intensity measurements, from multiple narrow-band wavelengths filter and scatter detectors. Each intensity measurement is known in the industry as a parameter. The flow cytometry is thus measuring 100 different parameters for each event in this case, for a total of about a million such events. The data is recorded202515949sequentially, in the same order for each parameter. As a consequence, the resulting dataset is always interleaved: [(Pl, P2, ... , P100)i, (Pl, P2, ... , P100)2, (Pl, P2, ... , P100)3, ... ], where the first 100 measurements are for event 1, the subsequent 100 measurements are for event 2, and so on, as denoted by the event index 1,2,3,... .

[0196] The problem with interleaved data is that subsequent measurements, representing different parameter values, are largely unrelated. For example, Pl may represent the intensity of emitted light at 550 nanometer, from a specific fluorophore, while P2 may be representing the intensity of scattered light perpendicular to the laser beam direction. There is no direct correlation between the Pl and P2 measurement values. The Pl intensity values, while varying to some extent across the cell or particle population, may all have high intensity values in a range greater than 100,000, while the P2 intensity values may all have low values, 100 or less across the entire dataset. Naively attempting to directly compress such interleaved datasets of many unrelated or at best weakly related alternating parameter values would clearly result in poor compression performance.

[0197] For example, FIG. 14 is a graph showing an example of how light emitted by dyes or fluorophores from cells in flow cytometry are emitted in ranges of wavelengths. In this example, three dyes or fluorophores attached to treated cells react to the laser excitation light. To detect the light intensity emitted by the three dyes or fluorophores, three narrow-band detectors are used (Chi, Ch2, Ch3). Each detector can only detect light in a narrow band centered around a particular wavelength, chosen to detect the anticipated dye emissions. For example, the third detector channel (Ch3) can only detect light in a third wavelength range (1403), the second detector channel (Ch2) can only detect light in a second wavelength range (1402), and the first detector channel (Chi) can only detect light in a first wavelength range (1401). To maximize the detection sensitivity, each detector wavelength is selected to be close to the peak intensity of a corresponding dye (it’s not always possible to have an exact match). The term narrow-band means that the widths of the detected wavelength ranges (1401, 1402, and 1403) are much smaller than the relative range of the emitted light of the dyes or fluorophores.

[0198] Because of the broad light emission range of each dye, a detector receives light not only from the dye it is meant to detect, but also some light from the other dyes. For example, the second detector channel (Ch2) is receiving light from both the first dye / fluorophore and the second dye / fluorophore; and the third detector channel (Ch3) is receiving some light from all202515949three dyes / fluorophores. Ideally we would like to have a one-to-one mapping with each detector only measuring one and only one dye. But this is not possible in practice, and the problem gets worse as the number of dyes / fluorophores and / or detectors increases. The end result is that the measured dyes intensity data is inherently mixed: the first detector channel (Chi) is additionally measuring light from second and third dyes (e.g., 10% of the third dye + 20% of the second dye + 70% of the first dye); the second detector channel (Ch2) is additionally measuring light from the third dye (e.g., 40% of the third dye + 60% of the second dye); and the third detector channel (Ch3) is the only true detector (e.g., measuring (100% of the third dye). The data has to be unmixed, to remove the inherent mixing, in order to avoid making incorrect detection results.

[0199] It should be noted that the light can come from any source and is not limited to laser light. A laser may be convenient in many flow cytometry instruments due to its ability to generate high intensity light with a highly focused narrow light beam. However, the light source may alternatively (or additionally) include a high-power light emitting diode (LED), tungsten lamp, etc.

[0200] FIG. 14 shows only three different dyes or fluorophore; however, typical flow cytometry involves the use of hundreds of dyes and / or fluorophores having different emission characteristics, not ordered by wavelength or other meaningful way. Therefore, the mixing problem can be especially apparent. This illustrates how compressing flow cytometry data may involve unmixing the data to provide adequate compression, as described herein.

[0201] As discussed, challenges of compressing flow cytometry data relate to the fact that amount of data can be very large and that the data typically includes mixed data. Measurement parameters in flow cytometry may include, for example, forward scatter, side scatter, horizontal scatter, vertical scatter, fluorescence intensity, and time-based measurements (e.g., time that each cell passes through light beam and / or duration of each cell that passes through light beam. A dataset may include data for hundreds of thousands to millions of cells. In addition, there may be measurements for hundreds of dyes and / or fluorophores, each associated with mixing problems.

[0202] Instead of using the raw data, according to some examples described herein, the data can be de-interleaved and un-mixed first, to eliminate the interleaving and mixing of unrelated parameter values, and then clustering related or similar measurement data together, of un-mixed data, using multiple criteria, and independently compressing each cluster separately. One aspect described herein is to identify correlated measurements, by clustering, and compressing the202515949correlated cluster data together, unmixed with other unrelated data, to achieve much higher overall compression.

[0203] FIG. 15 provides an example of one parameter of the parallel narrow-wavelength filter detectors, for a particular dye stain known as Live / Dead cell indicator, or cell viability dye. This fluorescent dye is reactive with cellular proteins, yet it cannot penetrate live cell membranes. The dye is excited by light at a wavelength of 488 nm (nanometer), causing it to emits light at around 617 nm.

[0204] In presence of live cells (1501) in FIG. 15(a), LIVE cell, the cell membrane (1502) is intact, as illustrated by a uniform grid of black circles on the cell surface. Since the dye is unable to penetrate the cell membrane, it can only react with few exterior surface proteins, resulting in low intensity values, as indicated by just a few reactive sites (1503).

[0205] In presence of dead cells (1504) in FIG. 15(b,) DEAD cell, on the other hand, the dye can penetrate the damaged membranes of dead cells (1504), illustrated by incomplete circles (1505), and react with both the exterior and interior proteins, resulting in significantly higher intensity values, as indicated by significantly many more reactive sites (1506).

[0206] The difference in fluorescence intensity between live and dead cells is typically larger than 50-fold, thereby providing complete simultaneous discrimination between live and dead cell populations. FIG. 15(c) illustrates the observed separation between the two populations. The chart shows a histogram of the number of cells encountered as a function of the observed Live / Dead dye intensity. The chart shows the distribution of live and dead cells is clearly bi-modal, evident by the two distinct gray peaks, with no overlap between, because the number of cells counted with intensities in a wide range near 103intensity is zero. The live cells intensities (1507) are clustered around 102intensity, while the dead cells intensities (1508) are clustered above 104intensity or higher.

[0207] Assume for example that parameter P3 is designated to measure the intensity of the Live / Dead dye. This means the narrow- wavelength filter D3 ( 1310) in FIG. 13 is tuned to pass the emitted light of the fluorescent Live / Dead dye, at about 617 nm, and block all other wavelengths. As described earlier, the flow cytometer output is comprised of interleaved data of the Pl, P2... , Pl 00 parameter values, in increasing event order. The interleaving intermingles unrelated parameter values, causing wild random intensity jumps from one observed intensity to202515949the next, therefore resulting in poor compression when attempting to compress the interleaved data directly with existing compression methods.

[0208] In the case of the Live / Dead cell P3 parameter, even after un-mixing, by gathering the observed P3 intensities for all events, and just looking at the aggregated P3 vector, independent of all other parameters, the P3 data still contains large jumps in observed intensity between events, because the live and dead cells are intermingled in the flow, it is never the case that all the live cells pass through the light first, followed by all the dead cells. On the contrary, the live and dead cells are uniformly suspended in the fluid, and may cross the light beam in a completely arbitrary random order. The observed intensity is bound to widely fluctuate between low intensity values for live cells, and high intensity values for dead cells. Large random fluctuations in the intensity values are inevitably bound to result in poor compression.

[0209] Based on the observed intensity behavior of the emitted intensity of this dye, as outlined in FIG. 15(c), it is reasonable to expect a classifier, based on the P3 intensity values, to divide the cell population in two: a population of live cells, and a population of dead cells. A method, using the classifier output, can split the P3 parameter data into 2 vectors: a vector P3A comprised of the observed live cells P3 intensity values, and a vector P3B comprised of the observed P3 dead cell intensities. A third 1 -bit index vector, with a single bit per entry, is also created with the same length as P3, the index entries are set to either zero or one depending on the corresponding P3 entry, whether is a live or a dead cell. In one example, the index vector is converted into a hop vector, which is used for reconstructing P3 during decompression.

[0210] Instead of attempting to compress the P3 vector directly, the classifier can be used to split the P3 vector into P3A and P3B, and the method proceeds to compress each split vector independently. The P3A vector may only contain live cell intensities with low values, around 102intensity. The P3A vector contains no random jumps between low and high intensity values. All P3A values are well behaved, with only low intensity values. They can be easily compressed. A subset representative value is selected from the histogram of live cell intensities (1507) in FIG.15(c), such as the most frequent value or the average value of the live cell intensities, etc. The majority of the observed live cell intensities occur near the subset representative value. Only the differences from subset representative value need to be encoded, and the code words can be personalized for the observed P3A range, from 0 to 120 intensity units.202515949

[0211] Alternatively, if the cells exhibit a trend, a predictor can be fitted to the data, and a series of corrections from predicted values can be encoded in the compressed stream, as described above.

[0212] Likewise, the P3B vector may only contain dead cell intensities, with high values, above 104intensity. The P3B vector contains no random jumps between low and high intensity values. All P3B values are well behaved, with only high intensity values. They can be easily compressed as well. A different subset representative value is selected from the histogram of dead cell intensities (1508) in FIG. 15(c), such as the most frequent value or the average value of the dead cell intensities, etc. The majority of the observed dead cell intensities occur near the subset representative value. Only the differences from subset representative value needs to be encoded, and the code words can be personalized for the observed P3B range, from 8000 to 14000 intensity units. It should be emphasized that each vector P3A and P3B are compressed independently, each with its own personalized code words, there is no sharing of code words between them.

[0213] Finally, the index vector is independently compressed. The index vector contains only two values: either zero or one. It can be compressed nicely, because the likelihood of finding at least a number of similar run patterns of zeros and ones is high. And the dimensionality of the index vector is reduced by a factor of 64 relative to the original P3 vector containing double precision floating point numbers. Alternatively, the index vector may be converted to a hop vector, and be compressed together with either P3A or P3B.

[0214] The P3A live cells intensity values are compressed, independently and separately from the P3B dead cells intensity values, each with its own set of personalized code words, and sent with the compressed index vector to the compressed output stream, independent of all other parameter values Pl, P2, P4,... , P100.

[0215] High compression can be achieved through the use of un-mixing the data; using a classifier to identify similarly related measurements based on several criteria, one such criteria being the distribution of parameter measurement values; split the data into multiple vectors of similar properties such as the intensity values; and separately compress each vector using a set of personalized code words, particular to each vector. Since all data samples within each group P3A and P3B are similar - either high or low P3 intensity - a much higher overall compression can be achieved.202515949

[0216] In the case of flow cytometry data, the classifier can identify common trends in the flow cytometry data because, although there can be hundreds of parameters being measured at the same time, some of the data can exhibit a trend even if some of the data does not. In general, the classifier enables the grouping of parameters by such trends (e.g., correlations), so that each group can be compressed separately.

[0217] We next turn our attention to the flow cytometry data representation. As indicated above, all light intensity measurements of all the parameters Pl, P2, through Pl 00 are conveniently saved as 64-bit double precision floating point numbers according to the IEEE 754 floating point standard. The entire flow cytometry data is comprised of a header, a small amount of metadata about the dyes and filter wavelengths being used, followed by millions of sequential events, each event containing over 100 interleaved parameter intensity values, the value of each measurement is saved in the IEEE 75464-bit double precision floating point format.

[0218] The accuracy of the IEEE 75464-bit double precision floating point representation is rigidly fixed based on the representation size (64-bit), and cannot be modified. For example, 64-bit double precision floating point numbers are represented with about 15.9 decimal digits of accuracy, and single precision floating point numbers are represented with about 7 decimal digits of accuracy. The input data, on the other hand, rarely comes in with the same accuracy (e.g. 15.9 decimal digits) to match the fixed IEEE 754 representation accuracy.

[0219] On the one hand, physical sensors and detectors have limited accuracy well below the 15.9 decimal digits of accuracy. For example, flow cytometer light intensity detectors have, at best, only 6 decimal digits of accuracy (one part per million). This means whenever a data is saved in the IEEE 754 double precision floating point format, it has to be extended to have the required 15.9 decimal digits of accuracy. Only the leading first 6 or 7 digits are significant, while the trailing 8-9 digits contain predominately noise, which is always present, in any detector, as well as thermal noise which is always present in electronic circuits.

[0220] On the other hand, if an input number has more decimal digits of accuracy than the IEEE standard allows, the input number cannot be represented as is. The input number has to be rounded to the closest available number in the representation. This rounding changes the trailing least significant digits of an input number.

[0221] FIG. 16 illustrates the IEEE 75464-bit double precision floating point representation of a decimal value. The observed value of 1,234,567.89012345 is well within range of flow202515949cytometer intensity data. The subsequent digits presented in FIG. 16 are examples for illustration purpose. Any other selection of arbitrary digits exhibit similar behavior.

[0222] The IEEE 75464-bit double precision floating point representation encodes any number as a 64-bit quantity (1601), comprised of 3 parts: (1) A sign bit (1606), zero means a positive, one means a negative number; (2) 11 -bit exponent (1607) in the range of -1022 to +1023, expressed with a fixed offset of 1024; and (3) 52-bit mantissa (1608). The 3 parts together make a single 64-bit quantity.

[0223] Since the number of bits allocated to each of the three parts is fixed, the overall accuracy of the representation is fully determined and cannot be changed. Only certain numbers can be represented exactly, without any loss of information. Other numbers have to be rounded to the closest available representation in the IEEE 754 format. For example, the particular decimal value 1,234,567.89012345 cannot be represented in the IEEE 754 double precision standard, because an exact representation does not exist. Instead, the value has to be rounded to the closest available number in the IEEE 754 double precision representation, where closest defined as the representation with the smallest error from the actual value.

[0224] In this case, the input number is rounded to the closest IEEE 754 double precision representation (1602), with the smallest error, 1.23456789012344996444880962372E6.Compared with the original decimal value, one can readily see the first 14 leading significant digits are identical. However, the very last digit of 5 cannot be represented exactly. Instead, the digit 5 is rounded to the closest available sequence in the floating-point representation 4.996444880962372 (the relative decimal point added for reference only). The digits differing from the original value are underlined (1603) in FIG. 3. Granted, the rounded value is very close to 5, but is not exact.

[0225] These small rounding errors, when occurring, make it difficult to compress IEEE 754 floating point numbers. The value of the 15th significant decimal digit makes no practical difference on the flow cytometry outcome, whether the value is a 5 or a 4, because the device (a light intensity detector) is not able to resolve light intensity with such accuracy. At best, it is only accurate up to 10'6, equivalent to 1 parts-per-million. However, in terms of compression, a lossless compression method must preserve all the representation digits correctly, regardless of their decimal significance. The means the compression must encode the entire sequence (1602), including the rounded digits (1603, underlined) exactly as shown. The compression usually has202515949to spend many bits to preserve the exact rounding digits, even though the trailing leastsignificant digits carry no practical significance to the outcome.

[0226] The IEEE 754 double precision floating point representation is shown in (1601). The most accurate rounded value (1602) is encoded in 3 parts. The sign is zero since the number is positive (1606). The exponent (1607) value is 19, written as 1024+19 = 1043, with a fixed offset of 1024, and the mantissa bits are shown in (1608); white squares indicate zero, and black squares indicate a binary 1. The three parts together form a single 64-bit quantity (1609), according to the IEEE 754 format (gaps between the sign, exponent and mantissa parts are inserted for illustration purpose). The final result is shown in hexadecimal (base 16) representation (1604) as well as in binary format (1605), as two subsequent 32-bit binary words.

[0227] As one can readily appreciate, this type of data has nothing to do with ASCII sequences of text character strings, as currently assumed by many existing compression techniques. The rounding to the closest available representation creates additional bits that prevent long runs of regular byte boundary patterns. While any data can be viewed as a sequence of bytes, and the bytes interpreted as text characters, the actual data content in this case is a sequence of floating point numbers, each floating point number is represented by a sign bit, mantissa, and exponent part, according to the IEEE 754 floating point industry standard. A single change in one of the least significant bits of the mantissa, while numerically insignificant in practice, the byte in which it resides would altogether be interpreted as a different ASCII character by existing compression techniques.

[0228] Instead of simply truncating the trailing least-significant digits to zero, as most lossy compression methods do, the compressor chooses a convenient pattern to replace trailing digits, from among the current set of personalized code words. Since the trailing least-significant digits carry no practical information, the values can conceptually be replaced with any sequence, for lossy compression, without effecting the outcome. This method benefits from the following advantages: (a) Truncating to zeros is highly unnatural, since it is extremely rare to witness a number with an exact sequence of zeroes in the trailing least significant digits; (b) Truncating to zeros creates a sudden step function in the time or spatial domain, resulting in high-frequency harmonics in the frequency domain, which are more difficult to compress, thereby reducing compression; and (c) Instead, in a preferred embodiment, a readily available pattern is selected from the current personalized set of code words, the cost in terms of compression is minimal,202515949since it merely reuses existing code words. By setting a threshold well below the instrument accuracy, the original data accuracy is maintained.

[0229] FIG. 17 illustrates the dynamic range of the IEEE 754 industry standard double precision floating point numbers. The range of all the possible floating point numbers according to IEEE 754 is shown by dotted area (1701), including zero. Each double precision floating point number is represented with a sign bit, a 52-bit mantissa, and 11 -bit exponent part, altogether requiring 64-bits, or 8 bytes per double precision floating point number. The dynamic range of the data at full precision spans from ±2.23 x IO'308to ±1.80 x 1O+308, with approximately 15.9 decimal digits of precision per value, and with over 600 orders of magnitude in scale.

[0230] Flow cytometry devices, in general, are unable to measure light intensity with such level of accuracy. Even the best detectors with the highest sensitivity such as avalanche CCD arrays and electron photon multipliers can only measure light intensity with a limited accuracy. Under the best conditions, they can only distinguish among about 105to 106light intensity levels. This means the accuracy of these devices is limited to about 0.1 to 1.0 parts per million of the maximum full scale intensity level, that is, no more than 5 or 6 significant decimal digits, for any intensity level.

[0231] In many practical situations, including flow cytometry data, the observed input signal occupies only a small fraction of the vast dynamic range of the IEEE 754 double precision floating point representation, usually just a few orders of magnitude at most. For example, the range of most flow cytometer intensity data (1702) varies between 10'5and 10+7intensity units. In the example shown in FIG. 15(c), for instance, the range is even smaller, only from 10+1to 10+5units, just 4 orders of magnitude. While negative numbers are not natively produced by the array of narrow wavelengths filter and scatter detectors, negative numbers may still occur due to certain calculations such as the subtraction of background scattered light as means to compensate for reflections of light from the channel walls and optics imperfections. In many cases, negative numbers are ignored, truncated to zero. The range of negative values is generally smaller than the corresponding range of positive values. If not truncated, a second region of negative values may be present, spanning a negative range of -10'4to -10+6intensity units (vertical lines 1703). Overall, the range of both positive and negative flow cytometry data (1702 and 1703 combined) is just a tiny fraction of the IEEE 754 double precision range (1701), occupying a small portion of the entire vast space of all possible numbers, a sparse representation.202515949

[0232] Thus, the methods described herein are particularly suited to multidimensional ordered series data containing values that fall within a limited dynamic range within the number system representation of the original data, typically within several orders of magnitude, while the native number representation is generally capable of representing a considerably wider range of many more orders of magnitude. For example, the multidimensional ordered series data contain values that fall within a dynamic range of < 10 orders of magnitude, more preferably < 8 orders of magnitude, and even more preferably < 6 orders of magnitude. In FIG. 15(c), the dynamic range is limited to just < 4 orders of magnitude.

[0233] The IEEE 754 double precision standard is designed to support the broadest possible range of floating point numbers, from tiny 2.23 * IO'308to huge 1.8*10308, with 15.9 or more decimal digits of accuracy, for both positive and negative numbers. This range is large enough to support practically anything, from sub-atomic particles to astronomical outer-universe distances, and can be applied universally, to any data content.

[0234] However, the IEEE 754 double precision representation is rather wasteful when flow cytometry data is concerned, especially in terms of compression. The range of flow cytometry intensity values in practice is much more limited, typically varying between zero (no light intensity) to 4*107(maximum intensity value). While negative intensity values are not normally directly observed, they may rise out of calibration and compensation processes, such as subtracting the observed background scatter or intensity when no cells are present in the light beam. None-the-less, even when taking both positive and negative numbers into consideration, the overall practical range of flow cytometry intensity values spans just a tiny fraction of the possible IEEE 754 floating point number range. There are billions of billions (about IO301to be exact) of reserved combinations for possible intensity values which could never occur with flow cytometry data.

[0235] This means the IEEE 754 standard floating point number representation is effectively a sparse representation as far as flow cytometry data is concerned.

[0236] Instead of using the IEEE 754 standard for representing flow cytometry data, a classifier is used to split the data, as illustrated with the live / dead cell parameter P3 example in FIG. 15, and seeks to find a personalized individual representation, independently and separately for each clustered data, just large enough to cover the data at hand. Instead of using a universal representation that supports all possible numbers, an alternative personalized representation is202515949used. The personalized representation is a tight representation, designed to support only the set of values present in a current data subset, thereby drastically reducing the number of combinations that need to be supported for compression. The compressor has to include the alternative representation in the compressed data stream, in a compressed form, for the encoder to be able to interpret it correctly to restore the original subset data upon decompression.

[0237] The personalized representation can be optimized for each data subset content. For example, the personalized representation for the subset vector P3 A, comprised of live cell population of un-mixed P3 parameter values, in FIG. 15, only need to consider the compression of low intensity values < 120, a tiny fraction of the IEEE 754 representation range. Likewise, the personalized representation for the subset vector P3B, comprised of dead cell population, only needs to consider high intensity values between 8000 and 14000 intensity units. In this case, the personalized representations of the subset vectors P3A and P3B share no common values, because there is no overlap, no common values between the two subsets. The reduced range of each subset allows for a much tighter personalized representation, using fewer than 64-bits per value like the IEEE 754 standard representation.

[0238] Depending on the classification, if there is overlap between certain data subsets, the common parts, which may appear in multiple clustered subset representations, if any, may be included once at the beginning in a shared global dictionary, or simply tagged as such upon first occasion, instead of being repeatedly and multiply encoded with each independent subset. The shared elements of personalized representations are only need to be kept until the last occasion of use, tagged by reference subset, and can be discarded thereafter.

[0239] In the example of flow cytometry data containing live / dead cell data, for instance, a classifier is used for diving the cell population into two groups of live and deal cell clusters, based on the measured P3 parameter intensity values, where the live cells have low intensity values, while the dead cells have high intensity values. Within each clustered subset, a single value may be calculated as a personalized representation of the subset intensities. In some examples, the most frequent intensity value in each subset is selected as the personalized representation. The most frequent intensity value corresponds to the intensity with the maximum peak in histograms (1507) and (1508) in FIG. 15(c), of the two live and dead cells populations, respectively.202515949

[0240] In some examples, the average intensity value in each subset may be selected as the personalized representation.

[0241] During compression, the compressor encodes the personalized representation of each subset, only once, in the compressed stream. Within each subset, the compressor only needs to encode the difference of each intensity value from the subset personalized representation value. If the deviations from representative personalized value within a subset are small, there is likely to be a substantial compression gain, with repeating patterns, instead of attempting to compress the original intensity values directly into the compressed stream.

[0242] In some cases, if a trend or a pattern is detected in the observed intensity values of a particular cluster, a predictor may be fitted to the data, as described herein. The compressor encodes the predictor coefficients in the compressed data stream. Within each cluster, the compressor only needs to encode a stream of small correction values, for each data point, to correct for any difference from predicted value. When the prediction is close to the actual values, the entire compress stream essentially becomes just a side channel of small prediction error corrections in the compressed stream. The use of a predictor can yield large gains in compression when the observed data trends can be effectively predicted to a large extent, with only minor corrections encoded in the compressed stream.

[0243] In some examples, multiple different predictors may be fitted across the event index (n). For example, multiple event indexes (n) may be divided into a set of contiguous subsections, providing a contiguous sub-sequence range of the event index (n), starting at event and ending at event m, and providing a piecewise correlated prediction. For instance, a 1,000,000 cell index (n) can be divided into 10 pieces: the first 100,000 cells, the second 100,000 cells and so on, and fitting a different main parameter predictor (with different offsets and scale factors for the dependent parameters) in each section. This can improve the prediction accuracy in each section, instead of having to fit a single predictor across the entire index (n), which may be less accurate. In addition, this can address cases when multiple trends are encountered, for example, a rising trend at first, then changing into a decreasing trend.

[0244] The division of event indexes (n) into sections is not limited to using equal length sections of the event index (n). In other examples, the boundaries between sections may be chosen according to some predetermined criteria. For example, if a rising trend is observed at the beginning of a series of event index (n) values, and later changes into a decreasing trend, the202515949event index (n) may be divided into two sections (i.e., the first section with a rising trend, the second with a decreasing trend) and two different predictors may fit each section. This can provide more accurate predictions in each section separately, instead of using a single predictor throughout the entire index range, and forcing it to predict both trends simultaneously, which may not be as accurate in certain cases.

[0245] A block diagram of an example method of compressing a sparse multidimensional ordered series data is illustrated in FIG. 19. The input data, comprised of a sequence of indexed events, each event containing many, up to 100 or more, interleaved parameter values, in standard IEEE 754 floating point representation format, is first un-mixed by the data un-mixing block (1901). The purpose of un-mixing can be to de-interleave and un-mix the data to eliminate unrelated or weakly related parameter values from inter-mingling with other parameter values. The un-mixed parameter values are fed into a classifier (1902), to identify subsets of data sharing similar properties based on one or more selected criteria. For example, a classifier may be selected to identify subsets of data based on the intensity values, to identify ranges of data modality, such as low and high intensity ranges for live / dead cells. Other criteria may be employed to detect correlated parameter subsets in the data based on similar intensity ranges or other data properties, and / or to localize the data subsets based on proximity in time. Multiple criteria based on the properties of input data may be employed.

[0246] The classifier output (1902) is used for splitting the input data (1903) into multiple independent subsets of data (VI -Vk), of similar intensity values, that can be independently compressed. For example, the un-mixed live / dead P3 parameter data can be split into two subsets, P3A and P3B, comprised of the populations of live and dead cells, with low or high intensity values, respectively. In general, the un-mixed data is split (1903) into subsets of similar properties. The objective of partitioning the data into subsets is to re-order the data in a way that results in considerably reduced random variability within each subset, relative to the original data, in order to achieve better compression.

[0247] The resulting data subsets (1904), after splitting (1903), are compressed independently and separately of each other. The algorithm is suitable for parallel processing, since data subsets are independent, and can be compressed in parallel on different CPU cores or on the cloud.202515949

[0248] If a trend is detected in a particular data subset, a predictor (1905) may be used, by fitting a predictor to the observed subset data. If no particular trend is detected, a simple representative value may be selected for each subset, such as the most frequent value, or the average intensity, etc. as a subset representative value.

[0249] When a predictor is used, the predictor coefficients are compressed and encoded into the compressed stream. Otherwise, when a representative value is used, whether the most frequent intensity value or the average value, the representative value is encoded in the compressed stream. This effectively communicates the predictor coefficients or representative values to the encoder, on a subset-by-subset basis, to be available during decompression, in order to restore the original data values.

[0250] Regardless of whether a predictor, a representative value, or an alternative method is used, only the differences from predicted or representative values need to be encoded in the compressed bit stream. The differences usually have a lower range than the original values. In particular, when the prediction is good, only small minor corrections need to be encoded in the compressed stream, if any. For example, the dead cell population of the P3B subset have intensity values in the range of 8000 to 14000 intensity units. If a representative value of 11,000 is selected, for example, any difference is reduced to a range of -3,000 to +3,000 intensity units, and the likelihood of repeating code words is higher, yielding better compression.

[0251] For each data subset, a set of personalized code words is generated (1906) for the specific data pattern of the subset, whether predictor corrections or subset average differences. The code words are optimized for the set of intensity differences at hand, independent of the other subsets code words. At his stage, the subset data is no longer in IEEE 754 floating point format, it is converted to a set of fixed representations, optimized for the subset data, and entropy encoded (1907) to achieve good compression. Finally, the encoded information is streamed out to the compressed bit stream.

[0252] In one implementation, the method concentrates on the expected dynamic range of the observed signal, for example Ymn falls between 0 and 107intensity units, and discards all the excess bits of information associated with the representation of numbers outside the observed signal range. Depending on user choice, the compression can be entirely lossless, or, alternatively, the user can choose to have higher compression while accepting a small loss of accuracy, for example using 7 decimal digits instead of the maximum 15 decimal digits of202515949precision. For example, the accuracy with 7 decimal digits of precision would be better than ±0.1 parts per million of the observed intensity value, which is generally considered to be adequate in practice. Depending on the application, the user may trade off a little accuracy in order to obtain much higher compression.

[0253] Instead of attempting a loose interpretation of the data as ASCII text characters and relying on a largely unpredictable chance of finding long matching sequences of text characters like conventional compression techniques, the method, according to some examples, takes advantage of a variety of characteristics common to large data sets acquired by scientific and engineering equipment to achieve predictable and consistently superior compression.

[0254] First, instead of attempting to preserve the IEEE 754 floating point representation, the methods described herein can include identifying the observed dynamic range of the data, and discarding all the excess bits of information associated with the representation of numbers outside the signal range. In other words, encoding only the region 1702 in FIG. 17, where the observed data resides, instead of the entire much larger floating point region 1701.

[0255] Second, the method described herein is specifically designed for compressing floating point data. The method takes advantage of knowing the data structure, instead of attempting to treat the data as an arbitrary collection of ASCII text characters like existing compression methods.

[0256] Third, instead of recording hundreds of thousands of floating point numbers in turn for each event, the present method un-mixes the data, and uses a classifier to split the data into subsets of similar properties. Within each subset, the proposed method predicts the sequence of values, using either a local, a piecewise or a global predictor, and replaces the entire sequence of values with a small set of predictor coefficients, coupled with a small stream of local error corrections from predicted values. The ability to accurately replace hundreds of thousands of subsequent (non-linear) double precision floating point values (8 bytes each) with just a description of a predictor, e.g., a set of predictor coefficients is one way in which the present methods give rise to very high compression ratios that are not possible with conventional compression techniques based on random text pattern matching.

[0257] Fourth, the methods described herein may be specifically designed for compressing ordered series data that is sparse.202515949

[0258] Fifth, unlike existing compression techniques that independently and separately attempt to compress the input values, the method, according to some examples, takes advantage of the frequency of occurrence of both intensity and hop distances, and compresses the Cmndata intensity difference levels together with the incremental hop distance values.

[0259] Furthermore, the methods may be designed for high throughput compression of data in near constant lookup time, regardless of the amount of data. The decompression time is even faster, since most computations are done during compression, and the decompression is reduced to simple sequence of hash map lookup and write to the output buffer.

[0260] As mentioned above, sparse data is particularly suited to compression using the methods and systems described here. As used herein the terms “sparse” and “sparse data” refer to a data set in which only a small portion of the total number of points in the representation space are associated with meaningful signal of the actual sample being analyzed, with trailing least-significant digits of the data in the set is attributed to noise (e.g. electrical or chemical) which is always present, such as in the detector and analog-to-digital circuits, as well as due to contaminants and / or solvents that come with the material to be analyzed. The aim can be to leverage the “sparseness” of the data within the IEEE 754 representation for achieving higher compression than otherwise possible with existing compression methods.

[0261] In particular, sparse ordered series data can comprise a plurality of observed Ymnvalues, interspaced with intervals of quiet periods, where the data remains relatively stable throughout the quiet periods. The quiet periods may correspond to regions in which the values are at or below a minimal intensity level, primarily dominated by noise.

[0262] Yet another aspect of data sparseness is illustrated in FIG. 19. Instead of attempting to compress the Ymndata directly, the observed Ymndata values can be transformed into a sequence of relative incremental Cmndifferences and hops from predicted or representative subset values, and compressed into the resulting sequence. The combination of reduced intensity ranges and incremental hops, together with the ability to accurately predict the subset data with just a few coefficients, is what gives rise to better compression than otherwise possible with existing conventional compression techniques.

[0263] Any of the methods (including user interfaces) described herein may be implemented as software, hardware or firmware, and may be described as a non-transitory computer-readable storage medium storing a set of instructions capable of being executed by a processor (e.g.,202515949computer, tablet, smartphone, etc.), that when executed by the processor causes the processor to control perform any of the steps, including but not limited to: displaying, communicating with the user, analyzing, modifying parameters (including timing, frequency, intensity, etc.), determining, alerting, or the like.

[0264] When a feature or element is herein referred to as being “on” another feature or element, it can be directly on the other feature or element or intervening features and / or elements may also be present. In contrast, when a feature or element is referred to as being “directly on” another feature or element, there are no intervening features or elements present. It will also be understood that, when a feature or element is referred to as being “connected”, “attached” or “coupled” to another feature or element, it can be directly connected, attached or coupled to the other feature or element or intervening features or elements may be present. In contrast, when a feature or element is referred to as being “directly connected”, “directly attached” or “directly coupled” to another feature or element, there are no intervening features or elements present. Although described or shown with respect to one embodiment, the features and elements so described or shown can apply to other embodiments. It will also be appreciated by those of skill in the art that references to a structure or feature that is disposed “adjacent” another feature may have portions that overlap or underlie the adjacent feature.

[0265] Terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. For example, as used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items and may be abbreviated as “ / ”.

[0266] Spatially relative terms, such as “under”, “below”, “lower”, “over”, “upper” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if a device in the202515949figures is inverted, elements described as “under” or “beneath” other elements or features would then be oriented “over” the other elements or features. Thus, the exemplary term “under” can encompass both an orientation of over and under. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Similarly, the terms “upwardly”, “downwardly”, “vertical”, “horizontal” and the like are used herein for the purpose of explanation only unless specifically indicated otherwise.

[0267] Although the terms “first” and “second” may be used herein to describe various features / elements (including steps), these features / elements should not be limited by these terms, unless the context indicates otherwise. These terms may be used to distinguish one feature / element from another feature / element. Thus, a first feature / element discussed below could be termed a second feature / element, and similarly, a second feature / element discussed below could be termed a first feature / element without departing from the teachings of the present invention.

[0268] Throughout this specification and the claims which follow, unless the context requires otherwise, the word “comprise”, and variations such as “comprises” and “comprising” means various components can be co-jointly employed in the methods and articles (e.g., compositions and apparatuses including device and methods). For example, the term “comprising” will be understood to imply the inclusion of any stated elements or steps but not the exclusion of any other elements or steps.

[0269] In general, any of the apparatuses and methods described herein should be understood to be inclusive, but all or a sub-set of the components and / or steps may alternatively be exclusive, and may be expressed as “consisting of’ or alternatively “consisting essentially of’ the various components, steps, sub- components or sub-steps.

[0270] As used herein in the specification and claims, including as used in the examples and unless otherwise expressly specified, all numbers may be read as if prefaced by the word “about” or “approximately,” even if the term does not expressly appear. The phrase “about” or “approximately” may be used when describing magnitude and / or position to indicate that the value and / or position described is within a reasonable expected range of values and / or positions. For example, a numeric value may have a value that is + / - 0.1% of the stated value (or range of values), + / - 1% of the stated value (or range of values), + / - 2% of the stated value (or range of values), + / - 5% of the stated value (or range of values), + / - 10% of the stated value (or range of202515949values), etc. Any numerical values given herein should also be understood to include about or approximately that value, unless the context indicates otherwise. For example, if the value “10” is disclosed, then “about 10” is also disclosed. Any numerical range recited herein is intended to include all sub-ranges subsumed therein. It is also understood that when a value is disclosed that “less than or equal to” the value, “greater than or equal to the value” and possible ranges between values are also disclosed, as appropriately understood by the skilled artisan. For example, if the value “X” is disclosed the “less than or equal to X’ as well as “greater than or equal to X” (e.g., where X is a numerical value) is also disclosed. It is also understood that the throughout the application, data is provided in a number of different formats, and that this data, represents endpoints and starting points, and ranges for any combination of the data points. For example, if a particular data point “10” and a particular data point “15” are disclosed, it is understood that greater than, greater than or equal to, less than, less than or equal to, and equal to 10 and 15 are considered disclosed as well as between 10 and 15. It is also understood that each unit between two particular units are also disclosed. For example, if 10 and 15 are disclosed, then 11, 12, 13, and 14 are also disclosed.

[0271] Although various illustrative embodiments are described above, any of a number of changes may be made to various embodiments without departing from the scope of the invention as described by the claims. For example, the order in which various described method steps are performed may often be changed in alternative embodiments, and in other alternative embodiments one or more method steps may be skipped altogether. Optional features of various device and system embodiments may be included in some embodiments and not in others.Therefore, the foregoing description is provided primarily for exemplary purposes and should not be interpreted to limit the scope of the invention as it is set forth in the claims.

[0272] The examples and illustrations included herein show, by way of illustration and not of limitation, specific embodiments in which the subject matter may be practiced. As mentioned, other embodiments may be utilized and derived there from, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Such embodiments of the inventive subject matter may be referred to herein individually or collectively by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept, if more than one is, in fact, disclosed. Thus, although specific embodiments have been illustrated and described202515949herein, any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.

Claims

202515949CLAIMSWhat is claimed is:

1. A computer-implemented method of compressing a sparse multidimensional ordered series data, the method comprising:accessing the sparse multidimensional ordered series data comprising indexed data sets, each indexed data set comprising a set of variables (ymn), wherein n is an event index representing an event, and m is a parameter index representing a parameter, wherein each event index (n) is associated with multiple parameter indexes (m); partitioning the sparse multidimensional ordered series data into subsets of data by:un-mixing the set of variables (ymn), to reduce cross-parameter dependencies introduced by the measurement process, thereby providing un-mixed data; classifying, using a classifier, the un-mixed data based on one or more desired criteria; andsplitting the un-mixed data into the subsets of data, wherein each subset of data is comprised of a set of parameter indexes (m) of one or more parameters; for each classified subset of data:predicting, using a predictor, a value for each variable (ymn) as a function of the event index (n) based on a trend observed in the classified subset of data, wherein a representative value is selected as the value when a trend is not observed in the classified subset of data;calculating a series of corrections (cmn) for each variable (ymn) from the predicted value;calculating a hop offset value (Amn) for each correction (cmn) of the series of corrections (c mn ); andgenerating a compressed output, the compressed output comprising a decoder legend configured to decode the predictor, the series of corrections (cmn), and the hop offset value (Amn) for each classified subset of data.

2. The method of claim 1, wherein calculating the series of corrections (cmn) for each variable (ymn) comprises assigning an intensity code word to each correction (cmn) based on a202515949value and a frequency of the correction (cmn), wherein calculating the hop offset value (Amn) for each correction (cmn) comprises assigning a hop code word to each hop offset value (Amn) based on the value and frequency of the hop offset value (Amn).

3. The method of claim 2, wherein the compressed output further comprises code word data comprising the intensity code word and the hop code word for each correction (cmn) and each hop offset value (Amn), respectively.

4. The method of claim 1, wherein the decoder legend comprises a description of the predictor, a reverse intensity code word dictionary associated with the correction (cmn), and a reverse hop code word dictionary associated with the hop offset value (Amn).

5. The method of claim 1, wherein the subsets of data are identified by a classifier.

6. The method of claim 1, wherein the sparse multidimensional ordered series data is in single and / or double precision floating point format.

7. The method of claim 1, wherein the sparse multidimensional ordered series data contains values that fall within a dynamic range of less than 10 orders of magnitude.

8. The method of claim 1, wherein the sparse multidimensional ordered series data comprises a plurality of indexed x,y pairs.

9. The method of claim 1, wherein the predictor is a global predictor function.

10. The method of claim 9, wherein the global predictor is an nthorder polynomial function.

11. The method of claim 10, wherein the nthorder polynomial function is g(n) = ao +ai*n + a2*n2+ as*n3.

12. The method of claim 1, wherein the predictor is a piecewise predictor.

13. The method of claim 1, wherein the predictor is a local predictor.

14. The method of claim 1, wherein the predictor further comprises an error correction mechanism.20251594915. The method of claim 1, wherein the sparse multidimensional ordered series data is spectroscopic data.

16. The method of claim 1, wherein the sparse multidimensional ordered series data includes flow cytometry data.

17. The method of claim 16, wherein the parameters include one or more of: forward scatter measurements, side scatter measurements, fluorescence intensity measurements for each fluorescence channel, and time and / or duration each cell passes through light beam.

18. The method of claim 1, wherein predicting, using the predictor, the value for each variable (ymn) as a function of the event index (n) comprises:identifying a correlation among the set of parameter values (m) of a first classified subset of data having similar trends or lack of a trend as a function of the event index (n); determining a main (ymn) parameter from among the set of correlated parameter values (m), one of the set of correlated parameter values (m) or a combination of several correlated parameter values (m), according to a desired criteria;predicting, using a predictor, a value for the main parameter (ymn) as a function of the event index (n) based on a trend observed in the classified subset of data, wherein a representative value is selected as the value when a trend is not observed in the classified subset of data;determining one or more dependent correlated parameter values (m) using offsets and scaling factors associated with the identified correlation; andpredicting the values of each variable (ymn) of dependent correlated parameter values (m) in classified subsets of data using the one or more offsets and scaling factors.

19. The method of claim 1, wherein predicting, using the predictor, the value for each variable (ymn) as a function of the event index (n), includes dividing a plurality of event indexes (n) into a continuous sub-sequence range of the event index (n), starting at event and ending at event m, providing a piecewise correlated prediction.

20. A system for compressing sparse multidimensional ordered series data, the system comprising a non-transitory computer-readable medium with instructions stored thereon, that202515949when executed by a processor, cause the processor to:access the sparse multidimensional ordered series data comprising indexed data sets, each indexed data set comprising a set of variables (ymn), wherein n is an event index representing an event, and m is a parameter index representing a parameter, wherein each event index (n) is associated with multiple parameter indexes (m); partition the sparse multidimensional ordered series data into subsets of data by:un-mixing the set of variables (ymn), to reduce cross-parameter dependencies introduced by the measurement process, thereby providing un-mixed data; classifying, using a classifier, the un-mixed data based on one or more desired criteria; andsplitting the un-mixed data into the subsets of data, wherein each subset of data is comprised of a set of parameter indexes (m) of one or more parameters; for each classified subset of data:predict, using a predictor, a value for each variable (ymn) as a function of the event index (n) based on a trend observed in the classified subset of data, wherein a representative value is selected as the value when a trend is not observed in the classified subset of data;calculate a series of corrections (cmn) for each variable (ymn) from the predicted value;calculate a hop offset value (Amn) for each correction (cmn) of the series of corrections (c mn ); andgenerate a compressed output, the compressed output comprising a decoder legend configured to decode the predictor, the series of corrections (cmn), and the hop offset value (Amn) for each classified subset of data.

Images