Feature Request: Duplicate Finder

I have a massive library and unfortunately overtime it has inherited many duplicates. It would be fantastic to be able to search for duplicates and have the simple tools to mark the preferred version and deleted the duplicate.

15 Likes

I agree to this feature request. I have already faced this problem (some of you may have already encountered this problem when importing some photos received via Whatsapp.) and opted to look for duplicates before importing. I used a freeware software which is quite smart and efficient and you could use it to scan your mylio catalog folders.

1 Like

Hi Dewydd, what freeware software did you use and how did you get Mylio to respect your changes on disk? will Mylio not just download the deleted version from the Vault via a sync as it sees it as just missing?

Hi @Darkin100, the software is called Visipics, being a bit outdated might not be powerful enough for the volume of data we manage today but might be worth to give it a try.
About the interaction with Mylio, as I said, I did it before import. Now, you could ensure the device is isolated before opening Mylio, look for the “missing originals” (there is an indicator for it in the device’s status) and delete the items in that search results.

1 Like

At this time, Mylio only has an “exclude duplicates” feature during Copy Import (http://mylio.com/support/import-copy-folder/). Mylio’s algorithm for detecting duplicates is a combination of factors including “file hashing” as well as metadata comparisons. If 2 files are the same but have different filenames, one will still be omitted. We have a more complete library deduplication tool in the works but it is not fully developed yet. I’ll be sure to add your voice to the cause!

While we do not have any affiliation with any de-duplication tools, there are many out there that can make quick work of scanning your “Mylio Pictures” folder (or any other folder). One free application that other users have reported good success with is called VisiPics. We have no association with them, just thought you would like to know that it exists as an option.

When running a de-duplication application directly on your file system, be sure to exclude the “Generated Images.bundle” folder. This is where Mylio stores Thumbnails and Previews.

Mylio’s Folder View (http://mylio.com/support/folder-view/) is a direct representation of how the folders and files are stored in the file system of the computer and external drives. Before removing the duplicates from the file system, please be sure to disable the Safe File System Delete (https://mylio.com/support/safe-delete/) feature in the Source Folder Settings.

Safe Delete is a feature that prevents accidental deletes made in the file system from replicating out to other devices, resulting in the photos/folders being deleted everywhere. If you want Mylio to notice the changes you make outside of Mylio and update itself to match whatever changes you make, you can disable Safe Delete. The toggle for Safe Delete can be found by opening the View Menu (3 dots in top right corner), then selecting “Settings”, then “Source Folders”. We have some more information on our Support Site here:

7 Likes

Great.
My opinion about Visipics is that it’s great because it’s able to perform visual matching. So even photos which have been resized, downsampled, jpeg compressed, rotated could be spotted (cases in which the file’s “fingerprint” on disk is totally different)
Thanks

I am writing once more after I tried doing some sweeps with VisiPics last night. It is still efficient with photos having dimensions of 16Mpx. It can be set to spot exact duplicates or loosen the match to spot unnecessary photos coming from image bursts or similar close shots.
Just be careful that Mylio is in a good state before turining off Safe File System to avoid deletion of elements which were missing for some reason before the duplicates cleanup.

Thanks for the update @dewydd. I have started playing around with VisiPics and getting some decent results and will work to help me sort out my library, but the process is going to be laborious!!! What I would like to see is that as Mylio builds its index of images it builds a thumbprint of the image which is can then use to find duplicates instead of having to rescan file files every time. It would make things a lot faster. Then you could have a rebuild index task for doing a full scan when needed.

Hope for Advanced deduplication one day is why I bought Mylio in the first place.
I Imagine to have Mylio on my iPad on the commuter Train and decide how to handle duplicates.
I imagine that Mylio proposes an Image which has duplicates and I choose a) which is my New Original (best resolution), which Metadata, where to put the new Original in Folder, Album and b) which edited copies I want to keep where.

4 Likes

I completely agree with this request. Having thousands of rescued photos without metadata it is an indispensable feature I am really missing.

1 Like

This is probably my biggest challenge and I am continually working to identify my duplicates and keep the best one. I will try the freeware option while I wait for the advanced capability Mylio is working on (hopefully with a sense of urgency given this thread is getting a lot of support). I would caution against doing the cleanup outside Mylio, as disabling the safe delete has caused significant issues for me in the past. Instead, it is safer (in my experience) to search within Mylio using the “file:” syntax in the search bar and delete the unwanted duplicate from the search match list.

1 Like

Photosweeper has been amazing for duplicates so I don’t feel like I’m missing it from Mylio. It has several options, you can choose how alike the photos should be, and it definitely compares the photo and not just the file name.
I think I’ve been having trouble lately because sometimes making file changes, it seems to be better if Mylio is open and sometimes better if Mylio is closed.
Someone will have to inform me of the proper way but I THINK lately, it’s been the case that Mylio should be closed when using Photosweeper to delete duplicates because then Mylio when applying file changes can dump the .xmp file and not cause it to think there’s Lost Originals.

2 Likes

I am so nervous to use any photo duplicate remover outside of Mylio as I’m afraid I’ll lose some photos. Can’t wait for Mylio to have a duplicate finder.

Especially one that can find lower resolution duplicates and remove them.

These reasons are exactly why I use PhotoSweeper. You don’t even have to delete the duplicates, Photosweeper can “reveal in finder” and you can tag them. Or Photosweeper can simply move the duplicates to another folder.
It also has settings you can designate for it to automatically mark the worse of the duplicates - file size is one of those options.
I realized recently whether Mylio is open or closed, if you delete these files, Mylio will count them as lost originals but Mylio will also only show .xmp files and I’ve used this process enough that I know it was all the photos I recently deleted.

I found it very helpful to use VisiPics BEFORE importing to Mylio.

1 Like