Deduplication for HDD

[Awesomefreeman]

Does it make sense to use deduplication for HDDs? In the manual, it's marked "dangerous". Synology also does not allow it to be used for hard drives. What are the dangers of using this script for HDDs?

[007revad]

Without knowing why Synology decided not allow it to be used for hard drives I decided to mark running deduplication on HDDs as dangerous. e.g. At your own risk, and don't blame me if something goes wrong.

Maybe Synology have not tested it or did test it and decided not to implement it for some reason. The reason could be they thought it takes too long.

I've tested it on a HDD and it worked as it should. People do use this script to deduplicate HDDs and none have reported anything bad happening.

[ThomasGoering]

One more reason for Synology not allowing it for HDDs could be:

Because data deduplication removes duplicate data blocks, it can make some data less contiguous and affect the read-write performance.

Accessing fragmented data affects the performance of HDDs more than SSDs. And this is something I can live with for my HDD volumes that are used for archiving and backups.

[MaximumFish]

Just looking to enable dedupe myself and came across this (awesome) script, and wondered the same thing. Performance limitations are a concern in busy environments, but I'd imagine it becomes far less of an issue with an SSD based read/write cache attached to the volume. Even less so in typical home environments.

That said, the amount of random access reads and writes that happens in order to do the initial dedupe could be cause for concern if the health of the drive(s) are in any doubt. Drives die during RAID rebuilds and I'd assume it's a similar amount of stress during a large dedupe job.

[007revad]

RAID rebuilds create a lot of reads and writes, and file hashing.

Deduplication just does a lot reads, but very little writes and no file hashing. For any file blocks that are the same between 2 files it just changes the duplicate block's reflink to point to the identical block, then marks the duplicate block as unused.

[MaximumFish]

Fair enough, I stand corrected. Though from experience (albeit not on a Syno NAS), the amount of time it takes to do the initial run still seems like it would be somewhat stressful for the drives.

Either way, I'm still going to use it myself! The amount of space saved can be quite incredible.

[cb12tre]

I would like to try enabling deduplication on two HDDs in SHR mounted on a 920+. If I experience slowdowns and later decide to disable deduplication using the command 'syno_enable_dedupe.sh --restore,' will the already deduplicated data remain accessible and usable as normal?

Is it possible to reverse the deduplication process? I’m not sure if there's a specific term for this, but I essentially mean duplicating the deduplicated data to return it to its original state

[007revad]

You can disable deduplication by unticking it in the volume's options.

The script's --restore option only restores the changed DSM settings, so all the already deduplicated data will remain accessible and usable as normal.

It is not possible to reverse the deduplication process.

[nowherenearaninja]

Depending on how much data it is and your other hardware, you could copy the data off of your NAS and then copy it back again to reverse deduplication