According to this invention, while the encoding efficiency of image data (e.g., a natural image) substantially maintains the conventional one, an image (e.g., a CG image or text document) having a small number of appearance colors is losslessly encoded at higher compression ratio. For this purpose, pixel data are input in the raster order and temporarily stored in a buffer. Pixel data at positions having undergone encoding are stored. A neighborhood matching determination unit generates first information representing whether a pixel having the same color as that of the pixel of interest exists in neighboring pixels a, b, and c, and second information for specifying whether a pixel having the same color as that of the pixel of interest exists, and if the pixel having the same color exists, specifying the neighboring pixel. A pixel matching detection unit counts the number of colors contained in the neighboring pixels a, b, and c, and generates information representing whether the number of colors is two or less, or three or less. On the basis of the first information and second information, a code generation unit outputs one or both of encoded data from a matched-pixel position encoding unit and prediction error encoding unit.