Problems with large archive

Janusz

After testing the program on a relatively small archive I tried to use it on my real large archive and encountered several problems. The main one is that while the program starts “Cheking for changes” (it does this automatically, at least do not start it intentionally) the user interface practically freezes. I use a desktop running Debian stable: Intel® Core™ i7-8700 CPU @ 3.20GHz, 16 GB RAM, the archive on an external drive connected through USB 3.
Is this a known problem? What do you suggest?

Andrey

Janusz no, this is not a known problem. If interface freezes, then database access probably is the bottleneck, not a CPU. Do you run on SSD?

Janusz

Andrey Yes, Linux is installed on a single SSD partition. BTW, the archive consists mostly of Google Photos Takeout, which includes a very large directory “Archiwum” with 19509665 files. Can this be a problem? One more information: the initial scanning took about 10 hours.

Andrey

Janusz BTW, the archive consists mostly of Google Photos Takeout, which includes a very large directory “Archiwum” with 19509665 files. Can this be a problem?

Wow, I never tested it with such a large folder. Yeah, I guess that could be the reason. Can you please split is up to several smaller folders and try again?

Janusz

No progress for over 40 minutes despite there are some tonfotos processes running all the time. By now UI is frozen over 6 hours.
The status bar is not updated, but messages like those below appear systematically on the console:
synchronizing dossier 1699788430917 4Facebook
scanning 1699788430917 4Facebook

Janusz

Today I tried to reproduce the program with version 1.6.7 (free) on Windows 11 Home (the same computer which is used for Linux but a standard HDD drive). The scanning time estimation was 8 hour, the program crashed without any error message after about 4 hours.

Janusz

I removed the photos leaving only one in the directory “Archiwum” and it seems now to work (I think I really don’t need the archive directory as it just contains the photos moved to appropriate albums).

twuble

I have the same problem, but with folders that are about 50-100,000 photos in size.
I had a folder of 100k and it would take a long time to even show any photos, scrolling down the list was also very slow, when i split them into folders of 10,000 then they loaded a fair bit quicker.

However the initial scanning phase of Tonfotos each time i run it always seems to take ages and the app often freezes while do this if i have a large library selected, even after all the photos have now been scanned/analysed.

Clicking between the tabs (Folders/Map/People) etc always seems to take ages and trying to scroll down the photo list of large folder is very slow/freezing too.

(PC with Intel® Core™ i7-4790K CPU @ 4.00GHz, 4001 Mhz, 4 Core(s), 8 Logical Processor(s) and 16GB of RAM)

Andrey

twuble I guess 10.000 per folder is still too much for application to swallow, that is why some database requests just take too long when folders are that big. Overall, program is capable for working with several terabytes without delays, but if you make folders that large that seems to be a problem for it.

GeertBE

Do you mean 10.000 images in 1 single folder on the HD,
Or a folder with subfolders with more than 10.000 images?

Like 10.000 images in a folder selected with “photo library locations”
So a folder selected with subfolders that contains more then 10.000 images

Andrey

GeertBE Subfolders are no problem at all, and 10 000 is not that much actually. However putting all images in one folder causes issues, and not only for Tonfotos, many applications will have issues with that. However, even in that case, 10 000 is not that much and Tonfotos handles it well, but 100 000 can be an issue.