Data set streaming and processing in computational storage devices

Abstract

This application is directed to data buffering in a computational storage device (e.g., a memory device having data processing capability). A memory device includes one or more processors, a volatile memory, and a non-volatile memory storing a data set. A portion of the volatile memory is allocated to facilitating data processing, and further includes a buffer having a buffer size smaller than a data size of the data set. The memory device loads a subset of the data set from the non-volatile memory to the buffer, and identifies a first data portion having a predefined portion size. In accordance with a determination that the first data portion has been processed, the memory device loads a next data portion of the data set from the non-volatile memory to the buffer in place of a subset of the first data portion (e.g., less than all or all of the first data portion).

Images