My server storage started getting full and I started looking into what was taking up so much space. I thought that I knew what was happening but actually had no clue.

My homelab server is running Unraid and 50TB of total storage. It has went through a few hardware revisions over the years and many HDD upgrades/additions.

My understanding was the media was placed into a directory after acquisition. Then, it was hardlinked to the directory Plex watches for media. This was not the case.

The problem was that I had an Unraid share for the acquisition directory and then another one for Plex to look at. I’ve come to realize and learn that hardlinking can not happen across Unraid shares. Whether this changed over the years or never has worked, I am not entirely sure. I think multiple shares were configured like this because of defaults. A guide I followed when I initially built the setup could have also influenced this configuration.

So now a lot of my media library is taking up double the space. How do I fix it? Using this Reddit post I was able combine the acquisition directory and the media directory where Plex watches for media.

Now that the duplicate files are all in a directory where they can be hardlinked, I used this utility to scan all the media. Its purpose is to find duplicate files and give you a list. Czkawka is packaged nicely by jlesage in the Unraid apps directory.

Scanning these files is heavy on the resources of the server as it hashes the files to compare them. I used the duplicate file tool on blake3 instead of the similar video tool to compare the file bits and bytes. The similar video tool looks at the visuals of the video. I wanted to avoid false positives on TV shows with the same intros, etc. It took a couple days to scan ~11,000 files with an 8 core 16 thread processor.

Then the process of confirming all the matches in the list starts. I like this manual process of confirming the match and then hardlinking. I go page by page and watch the available space on the server increase.

Another good resource was Trash Guides. I found that some of it was useful for Unraid but some wasn’t.

Hope this helps someone in a similar situation.