Instance-Separable Semantic Image Codec for Selective Object Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image compression technologies fail to fully utilize the redundancy between images and their semantics, leading to unnecessary computation, transmission, and storage costs, especially in applications requiring only specific object-related parts of the image.
Innovation Solution
An end-to-end instance-separable semantic-image joint compression system that exploits redundancy by encoding and decoding instances independently, utilizing a neural network-based system with semantic graph encoding and entropy models to achieve lossy or lossless compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If image and semantics are compressed independently using existing technologies, then the compression process is simple and straightforward, but significant redundancy remains unexploited leading to higher bit rate
Solution Approach 1:
The patent merges image compression and semantic compression into a unified joint compression framework. The image encoder and mask encoder share architectural components and are trained jointly to exploit correlations between visual content and semantic information, enabling redundant information to be removed across both modalities simultaneously, thus achieving lower bit rate compared to independent compression
Solution Approach 2:
The patent employs universal entropy models that serve multiple functions: they model both image and semantic data distributions, perform entropy coding for both modalities, and adapt to different compression scenarios. This multi-functionality allows the system to handle the complexity of joint compression efficiently without requiring separate specialized components for each task
2Ease of operation
If full image decoding is performed to retrieve specific object semantics, then complete image and semantic information is obtained, but unnecessary computation and transmission costs are incurred
Solution Approach 1:
The patent segments the compressed bitstream into instance-specific components through instance-separable encoding. Each object instance in the image is encoded with its own semantic representation and associated image features, allowing the decoder to extract and decode only the bitstreams corresponding to specific instances of interest, thereby avoiding the need to decode the entire image and achieving efficient instance-specific retrieval
Solution Approach 2:
The patent extracts instance-specific semantic information and associated image features during encoding, organizing them into separable representations. This extraction enables the decoder to selectively retrieve only the required instance data without processing unnecessary information, significantly reducing computation and transmission costs for applications requiring specific object retrieval
3Loss of information
If joint compression of image and semantics is implemented, then overall compression ratio improves and computational cost is reduced, but the system complexity increases
Solution Approach 1:
The patent employs parameter sharing and regularization techniques to control model complexity. By sharing weights between image and mask encoders and applying appropriate regularization, the system achieves effective joint compression while preventing overfitting and managing the complexity of the unified model, thus balancing compression performance with system tractability
Data Source
AI summary
An end-to-end instance-separable semantic-image joint compression system and method, comprising: image encoder and decoder, mask encoder and decoder, union encoder, embedding extraction module, semantic graph encoder and decoder, entropy coding module, union entropy model, image entropy model and mask entropy model. This invention can fully utilize the redundancy between images and their semantics (including instance masks, bounding boxes, categories and relationships) to improve rate-distortion performance; at the same time, this invention allows instances to be independently coded, facilitating the decoder side to retrieve the semantics and image of a specific object.