File Deduplication for Antivirus Scanning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large organizations face significant challenges in efficiently scanning data across numerous computing devices for viruses and malicious software due to the time-consuming nature of regular scanning and the presence of duplicate files, which can overwhelm devices and networks, wasting processing power.
Innovation Solution
Implementing a file-level deduplication system that calculates hashes for files across devices, identifies duplicates, and stores a single copy in a shared database, replacing multiple copies with links to reduce redundant scanning and optimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If regular antivirus scanning is performed on all files across all computing devices, then virus detection coverage is improved, but scanning time and processing overhead increase significantly
Solution Approach 1:
The patent merges multiple copies of identical files across different computing devices into a single representative copy for scanning purposes. By identifying duplicate files through hash matching and consolidating them, the system performs antivirus scanning on one copy instead of multiple copies, thereby maintaining comprehensive virus detection coverage while significantly reducing total scanning time and processing overhead.
2Reliability
If antivirus scanning is performed on duplicate files stored in multiple locations, then detection thoroughness is improved, but processing power is wasted
Solution Approach 1:
The system identifies duplicate files across multiple storage locations by calculating and comparing hash values. Once duplicates are identified, it consolidates them into a single representative copy that is scanned for viruses. This merging approach ensures that detection thoroughness is maintained (since the consolidated copy represents all duplicates) while eliminating redundant processing power consumption that would occur if each duplicate were scanned separately.
Solution Approach 2:
The patent creates a virtual copy or reference to the consolidated file copy that can be used by multiple computing devices. Instead of scanning the same physical file multiple times across different devices, the system uses a single scanned copy that serves all devices, thereby maintaining detection thoroughness while reducing processing power waste.
3Reliability
If all files are scanned individually without deduplication, then scanning completeness is improved, but network bandwidth consumption increases
Solution Approach 1:
The patent merges duplicate files from multiple computing devices into a single consolidated copy located on one device or central storage. By performing antivirus scanning on this single consolidated copy rather than individually scanning files on each device, the system maintains scanning completeness (all unique files are scanned) while dramatically reducing network bandwidth consumption that would result from transmitting and scanning duplicate files multiple times across the network.
Data Source
AI summary
Aspects of the disclosure describe methods and systems for performing an antivirus scan using file level deduplication. In an exemplary aspect, prior to performing an antivirus scan on files stored on at least two storage devices, a deduplication module calculates a respective hash for each respective file stored on the storage devices. The deduplication module identifies a first file stored the storage devices and determines whether at least one other copy of the first file exists on the storage devices. In response to determining that another copy exists, the deduplication module stores the first file in a shared database, replaces all copies of the first file on the storage devices with a link to the first file in the shared database, and performs the antivirus scan on (1) the first file in the shared database and (2) the files stored on the storage devices.