Image processing device and method of mobile device

Abstract

The present invention relates to an image processing device and method of a mobile device. The image processing device of a mobile device comprises: a memory storing image data; a processor for recording first image data of a linear address map method on the memory; and a converter for receiving the first image data to return second image data of a tile address map method.

Description

Image processing device and method for a mobile device

[0001] The present invention relates to an image processing apparatus and method for a mobile device, and more specifically, to an apparatus and method capable of using a linear address map and a tile address map together.

[0002] Generally, when an image processing application runs on a mobile device, system performance may be degraded due to DRAM bank crashes, cache misses, and Translation Lookaside Buffer (TLB) misses depending on memory access characteristics. To reduce these problems, a tile address mapping method may be used.

[0003] Figure 1 is an example of an image processing device using a traditional linear address map.

[0004] In a traditional linear address map (LIAM), the memory address of a pixel increases linearly along the horizontal axis.

[0005] LIAM allows for simple hardware design and is used as a default map in software.

[0006] A processor (100) stores an image in memory (200), an image processing device (300) processes it, and reads and displays it on a display device (400).

[0007] A linear address map allows the processor (100), image processing device (300), and display device (400) to store (write) image data in memory (200) in a linear manner and also to read (read) and process it in a linear manner, but depending on the characteristics of memory (200), it becomes a factor that causes a decrease in image processing performance.

[0008] More specifically, image processing applications often operate in blocks.

[0009] For example, convolution accesses memory by moving progressively in 3x3 pixel units.

[0010] However, in LIAM, addresses are continuous along the horizontal axis, but adjacent pixels along the vertical axis have very large differences in address values.

[0011] This can lead to reduced locality and unnecessarily access other memory pages repeatedly and frequently, which can degrade performance. To reduce unnecessary memory access and improve performance, a Tiled Address Map (TIAM) can be used.

[0012] Figure 2 shows an example of tile address mapping.

[0013] Figure 2 shows an example of a TIAM with 12 tiles.

[0014] The pixel addresses are arranged as indicated by the arrows. ImgH represents the number of pixels along the horizontal axis of the image, ImgV represents the number of pixels along the vertical axis, TileH represents the number of pixels along the horizontal axis of the tile, and TileV represents the number of pixels along the vertical axis of the tile.

[0015] In Figure 2, the numbers indicated by circle characters represent tile numbers.

[0016] The addresses inside the tiles are linear, and the addresses between tiles are arranged linearly in the order of the tile numbers.

[0017] Multiple masters operate simultaneously in the system, and when a tile is selected in the system, the selected masters use TIAM.

[0018] At this time, there was a limitation in that it was difficult to efficiently utilize the advantages of LIAM and TIAM because the address map could not be selected flexibly.

[0019] The problem that the present invention aims to solve, taking into account the problems of the conventional technology described above, is to provide an apparatus and method that can efficiently utilize the advantages of each address map by flexibly selecting a linear address map and a tile address map.

[0020] Another objective of the present invention is to provide a device and method capable of relatively high-speed processing by providing hardware that converts a linear address map into a tile address map.

[0021] In addition, the present invention provides an apparatus and method that allows the tile size to be easily adjusted according to the application when converting a linear address map into a tile address map.

[0022] An image processing device of a mobile device according to an embodiment of the present invention for solving the above technical problem comprises: a memory in which image data is stored; a processor that records first image data in a linear address map format in the memory; and a converter that receives the first image data and returns second image data in a tile address map format, wherein the processor can store the second image data in the memory.

[0023] In one embodiment of the present invention, the invention further includes an image processing unit that generates and returns third image data processed in a tile address map manner based on the second image data, the image processing unit converts the third image data into fourth image data in a linear address map manner and returns it, and the processor can store the third image data and the fourth image data in respective areas of the memory.

[0024] In one embodiment of the present invention, a display device is further included, and the processor can provide the fourth image data to the display device to display it.

[0025] In one embodiment of the present invention, the converter may include a coordinate extraction unit that extracts pixel coordinates of a linear address map; and a tile address calculation unit that converts the pixel coordinates extracted by the coordinate extraction unit into pixel addresses of a tile address map.

[0026] In one embodiment of the present invention, the tile address calculation unit receives tile size information and can convert the pixel coordinates extracted by the coordinate extraction unit into a tile.

[0027] In one embodiment of the present invention, the size information of the tile may be a hardware signal directly input to the converter.

[0028] In one embodiment of the present invention, the size information of the tile may be provided from the processor.

[0029] An image processing method for a mobile device according to an embodiment of the present invention, for solving the above technical problem, may include a method performed by a device comprising a memory, a converter, and a processor, the steps of: recording first image data in a linear address map format in the memory; inputting the first image data into the converter to generate second image data in a tile address map format; and storing the second image data in the memory.

[0030] In one embodiment of the present invention, the method may include: a step of generating third image data processed in a tile address map manner based on the second image data; a step of converting the third image data into fourth image data in a linear address map manner; and a step of storing the third image data and the fourth image data in respective areas of the memory.

[0031] In one embodiment of the present invention, the step of providing the fourth image data to a display device to display it may be further included.

[0032] In one embodiment of the present invention, the step of generating second image data may include: a step of extracting pixel coordinates of a linear address map; and a step of converting the extracted pixel coordinates into pixel addresses of a tile address map.

[0033] In one embodiment of the present invention, the conversion step to the pixel address can receive size information of the tile and convert the pixel coordinates extracted by the coordinate extraction unit into a tile.

[0034] In one embodiment of the present invention, the converter may further include the step of receiving size information of the tile as a hardware signal.

[0035] In one embodiment of the present invention, the converter may further include the step of receiving size information of the tile from a processor.

[0036] The present invention has the effect of preventing system performance degradation by allowing hardware or software elements related to image display to flexibly use a linear address map or a tile address map during the process of reading and writing image data to memory, thereby enabling the utilization of the advantages of each linear address map and a tile address map.

[0037] In addition, the present invention has the effect of improving the speed of tile address map conversion by configuring a converter that converts a processor's linear address map into a tile address map as hardware.

[0038] In addition, by controlling the converter using software or hardware signals to adjust the size of the tiles, it is possible to flexibly provide a tile address map of a size suitable for various applications.

[0039] Figure 1 is an explanatory diagram of a conventional linear address map method.

[0040] Figure 2 is an explanatory diagram for explaining the tile address map.

[0041] FIG. 3 is an example diagram of the configuration of an image processing device of a mobile device according to a preferred embodiment of the present invention.

[0042] Figure 4 is a block diagram of the converter.

[0043] FIG. 5 is an example diagram of the configuration of an image processing device of a mobile device according to another embodiment of the present invention.

[0044] To fully understand the structure and effects of the present invention, preferred embodiments of the present invention are described with reference to the attached drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various forms and various modifications can be made. The description of the embodiments is provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. In the attached drawings, components are depicted enlarged from their actual size for convenience of explanation, and the proportions of each component may be exaggerated or reduced.

[0045] Terms such as 'first' and 'second' may be used to describe various components, but said components should not be limited by said terms. These terms may be used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, 'first component' may be named 'second component,' and similarly, 'second component' may be named 'first component.' Furthermore, singular expressions include plural expressions unless the context clearly indicates otherwise. Unless otherwise defined, terms used in the embodiments of the present invention may be interpreted in the sense commonly known to those skilled in the art.

[0046] Hereinafter, an image processing device and method for a mobile device according to an embodiment of the present invention will be described in detail with reference to the drawings.

[0047] FIG. 3 is a block diagram of an image processing device of a mobile device according to a preferred embodiment of the present invention.

[0048] Referring to FIG. 3, the present invention may be configured to include a processor (10), an image processing unit (20), a display device (30), a converter (40) that converts a linear address map of the processor (10) into a tile address map, and a memory (50).

[0049] The image data stored in the memory (50) may be image data captured through a camera module, but the present invention is not limited to the device that acquired the image data. That is, it may include images downloaded from a network or copied (moved) from another device, rather than images captured by a mobile device through a camera module.

[0050] The processor (10) can execute an application to check image data in memory (50) and also perform editing of the application.

[0051] In this case, image editing includes various image processing such as rotation, resizing, and color conversion. Although the following description is based on image rotation, it will be readily understood by those skilled in the art that the present invention can be applied to various types of image editing.

[0052] The processor (10) can store the application's data in memory (50).

[0053] Typically, applications (software) running on a processor (10) use a traditional linear address map (LIAM).

[0054] The converter (40) converts the linear address map of the processor (10) into a tile address map and stores it in memory (50).

[0055] The converter (40) may include a physical circuit that converts a linear address into a tile address in response to a hardware signal of the processor (10).

[0056] The converter (40) can convert a linear address map into a tile address map by specifying the tile size by a software signal by the processor (10) or a hardware signal that sets the converter (40).

[0057] An example of a converter (40) is shown in FIG. 4.

[0058] Referring to FIG. 4, the converter (40) may consist of a coordinate extraction unit (41) and a tile address calculation unit (42).

[0059] The coordinate extraction unit (41) takes the linear address map of the processor (10, application) as input and extracts the coordinate values ​​of each pixel of the linear address map.

[0060] To do this, the base address (BaseAddr), information on the number of pixels on the horizontal axis (ImgH) and vertical axis (ImgV) of the image to be saved, and the number of bytes of the pixel (BytePixel) are received as input, and the pixel coordinate value (Coordinate (A, B)) is obtained.

[0061] At this time, the coordinates (A,B) from the linear address can be obtained using the following mathematical formula 1.

[0062]

[0063] In the mathematical formula 1 above, LIAMAddress is the coordinate of the linear address map, BaseAddr is the base address, ImgHB is the number of bytes of pixels on the horizontal axis of the image, and BytePixel is the number of bytes of pixels.

[0064] Next, the tile address calculation unit (42) calculates the tile address using the extracted coordinates.

[0065] To this end, the tile address calculation unit (42) can receive tile size information. The tile size information may be the number of pixels on the horizontal axis of the tile (TileH) and the number of pixels on the vertical axis of the tile (TileV).

[0066] At this time, base address (BaseAddr) information that serves as the standard for tile division can be received.

[0067] In other words, the pixel coordinates in the linear address map are reassigned to match the divided tiles.

[0068] A tile address contains multiple pixels horizontally and vertically, and square or rectangular tiles can be used as needed.

[0069] For example, the tile size can be determined to be 16x16 pixels. This is a widely used tile size in motion estimation, MPEG, and H.264 formats.

[0070] As a result of dividing the image into tiles, a tile address map can be generated and stored in memory (50).

[0071] That is, image data can be stored in memory (50) as a tile-shaped tile address map through a memory controller (not shown in the drawing) that stores data in memory.

[0072] The image processing unit (20, e.g., GPU) reads image data stored in memory (50) as a tile address map. That is, it reads the tiles and performs the necessary image processing, and after image processing, the image information is stored in memory (50) as a tile address map.

[0073] After image processing, the image information is also stored as a linear address map in another area of ​​memory (50).

[0074] The image processing at this time can be enlargement, rotation, etc. In the case of rotation, the image processing unit (20) can perform processing to read the tiles vertically and write them linearly horizontally.

[0075] The image processing unit (20) finally stores image data in memory (50) using a linear address map method, which can be understood as being due to the use of a linear address map in the display device (30).

[0076] The display device (30) reads and displays image data from memory (50) using a linear address map.

[0077] FIG. 5 is a block diagram of an image processing device of a mobile device according to another embodiment of the present invention.

[0078] Referring to FIG. 5, the present invention comprises a processor (10), an image processing unit (20), a display device (30), a converter (40) that converts a linear address map of the processor (10) into a tile address map, and a memory (50), wherein the function of the converter (40) can be turned on or off.

[0079] Here, the on or off of the converter (40) can be performed through hardware or software processing.

[0080] That is, as previously explained, when the processor (10) writes image data to memory (50) using the converter (40), the linear address map can be converted into a tile address map and used.

[0081] In addition, the converter (40) can be turned off so that when the processor (10) writes data to the memory (50) in the same way as the conventional method, it can be written in a linear address map manner.

[0082] That is, the present invention means that a linear address map method or a tile address map method can be flexibly selected and used in the processor (10) depending on the type of application supported.

[0083] At this time, the image processing unit (20) can read data from memory (50) in a linear address map manner, perform processing, and then store the data again in a linear address map manner.

[0084] The display device (30) reads and displays data in a linear map manner.

[0085] Although embodiments according to the present invention have been described above, they are merely illustrative and those skilled in the art will understand that various modifications and equivalent embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the following claims.

Claims

1. Memory where image data is stored; A processor that records first image data of a linear address map method into the memory; and It includes a converter that receives the above-mentioned first image data and returns second image data in a tile address map format, and The above processor is an image processing device of a mobile device that stores the second image data in the memory.

2. In Paragraph 1, It further includes an image processing unit that generates and returns third image data processed in a tile address map manner based on the second image data above, and The image processing unit converts the third image data into fourth image data using a linear address map method and returns it, and The above processor is an image processing device of a mobile device that stores the third image data and the fourth image data in respective areas of the memory.

3. In Paragraph 2, It further includes a display device, The above processor is an image processing device of a mobile device that provides the fourth image data to the display device to display it.

4. In Paragraph 1, The above converter is, A coordinate extraction unit for extracting pixel coordinates of a linear address map; and An image processing device of a mobile device comprising a tile address calculation unit that converts pixel coordinates extracted from the above coordinate extraction unit into pixel addresses of a tile address map.

5. In Paragraph 4, The above tile address calculation unit is, An image processing device for a mobile device characterized by receiving tile size information and converting pixel coordinates extracted by the coordinate extraction unit into tiles.

6. In Paragraph 5, The size information of the above tiles is, An image processing device for a mobile device characterized by being a hardware signal directly input to the above converter.

7. In Paragraph 5, The size information of the above tiles is, An image processing device for a mobile device characterized by being provided by the above-mentioned processor.

8. A method performed by a device including a memory, a converter, and a processor, A step of writing first image data of a linear address map method to the memory; A step of inputting the first image data into the converter to generate second image data in a tile address map method; An image processing method for a mobile device comprising the step of storing the second image data in the memory.

9. In Paragraph 8, A step of generating third image data processed using a tile address map method based on the second image data above; A step of converting the above third image data into fourth image data using a linear address map method; and An image processing method for a mobile device comprising the step of storing the third image data and the fourth image data in respective areas of the memory.

10. In Paragraph 9, An image processing method for a mobile device further comprising the step of providing the above-mentioned fourth image data to a display device to display it.

11. In Paragraph 8, The step of generating second image data is, Step of extracting pixel coordinates of a linear address map; and An image processing method for a mobile device comprising a step of converting the extracted pixel coordinates into pixel addresses of a tile address map.

12. In Paragraph 11, The above pixel address conversion step is an image processing device of a mobile device that receives tile size information and converts the pixel coordinates extracted by the above coordinate extraction unit into tiles.

13. In Paragraph 12, An image processing device of a mobile device further comprising the step of the converter receiving size information of the tile as a hardware signal.

14. In Paragraph 12, An image processing device of a mobile device further comprising the step of the converter receiving size information of the tile from a processor.

Images