Losing confidence in Mylio syncing

At first I was quite impressed with Mylio but the more I use it the more frustrations I am getting.

Case in point:
I performed a completely clean install of macOS 11 Big Sur and upon reinstall, Mylio would not recognize and stopped syncing to my NAS vault. It did however continue to sync to my external SSD. Contacting support, they said I would need to remove the NAS vault and add it as a new vault and let Mylio sync back up from the SSD. There was no ability to “adopt” the old vault. I backed up the vault and let Mylio do this.

Now my vaults are less than 800 MB but it still took Mylio a week to create the new NAS Vault. In addition the new value is only 327 GB. The backed up NAS vault is 767MB and my SSD vault is 778 MB.

In Mylio’s dashboard, it says my phone, my wife’s phone and our MacBook are In Sync as is my iMac and SSD Vault. The NAS vault has remained 17 to sync for days, yet it says to open with the SSD vault or my wife’s phone to continue syncing. This is while the dashboard says they are open and in sync.

I do not understand why the newly created NAS vault is 1/2 size of the other vaults and also why Mylio will not complete syncing of the last 17 photos.

Meanwhile it also says I have 8 unprotected media, 24,798 lost originals and 89 unsupported media. I think the lost originals was as I was loading I was also deleting a lot of duplicates so that may not be critical but this does not inspire confidence.

Is there any reason why one vault would be 1/2 size of the others when Mylio says it is “mostly” in sync?

I don’t really like the NAS vault in mylio. It’s basically a folder on a networked drive, and SMB / NFS drives are notoriously unreliable. It’s one reason why I want a headless linux version of mylio you can run on a NAS, that way you have very reliable syncing like you had a desktop on all the time running mylio.

What has worked for me reliability wise is a desktop computer, cloud drives (with encryption!) like amazon drive and my mobile phones. You can recreate the NAS experience by dedicating a small low power windows PC or mac mini to always be on running mylio in the background. This consumes a bit more background power, but with the new M1 mac mini it’s actually about the same amount of power consumption of a typical NAS!

What I like about cloud drives is they act as an offsite backup. If your external drive and desktop computers all get destroyed in a house fire, you’ll still have all of your photos. I might consider that to solve your problem and add a robust backup.

If you were using the file system to delete duplicates, it’s pretty likely that’s where the lost originals came from, in my experience (don’t blindly take my word for it obviously). But anytime I run a dedup program outside of Mylio, I end up with “lost originals” because the dedup program didn’t delete the .xmp files. If you look at these lost originals and the file ends in .xmp, I would assume that’s what’s going on.
I don’t use an NAS, so that’s as far as I can comment on the subject.

I’m covered for the back up. I have an offsite 24TB CrashPlan backup.

I wish there was a synology app that was a headless Mylio. That would solve several issues.

With cloud backup, say you need to share an original photo from an optimized phone, are you able to download the original while off the home network so you have access to the originals, over cellular? That could solve some issues as well.

I had a lot of issues with the XMP files getting orphaned as Photo Sweeper does not delete them then the photo is deleted. I now have a shell script that will delete orphaned XMP and AAE files to clean up the mess.

I would love to know what your shell script is. I’m just starting to get into that kind of stuff.

Yes, I’ve been able to download an original to my phone over cellular, with Amazon as the cloud drive it pulled from.

This is the shell script:

#!/bin/bash
while read sidecarFile 
do   
	Printf "Source: %s\n" "$sidecarFile"
	fileName=${sidecarFile%.*}
	
	lines=$(find "$fileName".* ! '(' -type f -iname '*.aae' -o -iname '*.xmp' -o -iname '.' ')' | wc -l)	
	if [ $lines -eq 0 ] 
	then
		Printf "  --  no match found  ******** DELETED ********\n" 
		rm "$sidecarFile" 
	else
		Printf "  --  matched\n"
	fi
done < <(find "$PWD" -type f -iname '*.aae' -o -iname '*.xmp' | sort -z)

The way it works is I have this saved as a file called “Remove sidecar.sh”. Open Terminal (this is for macOS), and change the working folder to the Mylio folder you want to remove orphans from. This will also do all sub folders in that folder. Then drag the “Remove sidecar.sh” file into terminal and press return. It will then execute the script.

For this to work in Mylio, it needs to be on a vault that is not optimized and with Mylio configured so that safe delete is off. As a precaution, I would test this on a small copy of your files before you run it on your main library so you get comfortable with it.

The way the script will work is that it looks for all .aae and .xmp files in the working folder (and subfolders) and for each one it sees if can match a file that is not a .aae or .xmp (ie, a jpeg, heir, raw, mov etc). If it does not find a match it will delete the sidecar. If it finds a match it leaves it and checks the next one.

Do not do this on a auto-optimized library. In that instance the sidecar files reside without the original photo as the photo is in a separate generated images bundle.

IF YOU RUN THIS ON AN AUTO-OPTIMIZED LIBRARY YOU WILL MESS IT UP.

1 Like

I asked Mylio about the right procedure to remove duplicates. Here is what Matthew said:

Hello Ted,

Thanks for the follow-up. Glad to hear you were able to get things restored.

When running a de-duplication tool directly on the folders that Mylio is using, you will want to disable the Safe Delete feature. Safe Delete is a feature that prevents accidental deletes made in the file system from replicating out to other devices, resulting in the photos/folders being deleted everywhere. If you want Mylio to notice the changes you make outside of Mylio and update itself to match whatever changes you make, you can disable Safe Delete. The toggle for Safe Delete can be found by opening the View Menu (3 dots in top right corner), then selecting “Settings”, then “Source Folders”. We have some more information on our Support Site here:

You may want to consider running a Windows VM on your Synology, and have that VM run mylio. See this post: https://community.mylio.com/t/nas-auto-scan-import/581?u=deinlakai