Disable folder scanning

Tommy

Ciao Andrey, I really appreciated the ability to turn off face and video scanning in the new version of Tonfotos.
It would be great to have also the possibility to disable folder scanning for new images after the first scan. In my case in fact I keep the photos in folders based on the year (with subfolders organized by date), and it doesn’t make sense to me that Tonfotos scans the 2021 (or 2020, or 2019 etc) folder every time. (I also didn’t understand if the scans take place every certain amount of time or only on opening…).

Andrey

Tommy you are right, scans are repeated on regular basis for number of reasons. However, if there are no changes, they are done pretty fast. And actually, they do not take any processing power nor do prevent you from working with application as usual, you can just ignore it. Why it is an inconvenience for you?

Tommy

Andrey while Tonfotos is scanning the cpu works hard and the fans start spinning (not sure if it’s because I’m on Linux or because my computer is quite old). Considering that I have collected 90000 photos in 19 years, I think it’s more useful and fast to scan only 3000 photos from the last year (and even in that folder I actually only add photos once a month, so I could disable the scan in that folder too…).

cpu

Andrey

I believe this is happening because your initial scan is not finished yet, face recognition and video indexing can be resource intensive tasks. This task needs to be finished once, otherwise photos will not show up in the index of Tonfotos. However, all consequent scanning will not be so resource intensive, they almost do not consume CPU.

Tommy

Andrey no no, the scan was successfully completed a few weeks ago, all photos are correctly displayed in Tonfotos.
Yesterday I noticed a certain acceleration by disabling facial recognition in each folder. But there’s still the problem of the spike in CPU usage for scanning new photos, which lasts about ten minutes at the start in my case, and then starts again during use of Tonfotos, causing lag in usage.
I still don’t understand if it’s a bug, so I thought it would be great to have the ability to disable the continuous scan, and then re-enable it only when needed.

Andrey

Tommy this does not sound normal. Usually scanning for changes does not take a lot of CPU. I guess it makes sense to figure out why in your case it does. Where do you store your photos? Is is local hard drive? Do you have enough memory in your system? What memory graph shows during this?

Tommy

Andrey photos are on local disk (not SSD), I have 8GB RAM and >100GB free disk space.

This is the CPU usage graph, around 50% (idle it hardly exceeds 20%).

And that is the terminal output.

$ tonfotos
checkpoint config: 6 ms
checkpoint app ready: 564 ms
[39045:1222/140724.821870:ERROR:sandbox_linux.cc(374)] InitializeSandbox() called with multiple threads in process gpu-process.
Server running at http://127.0.0.1:9871/
scanning 1669756668306 2017
Auto update URL: https://tonfotos.com/distribution/tonfotos/release/linux/x64
intercepted file:///usr/lib/tonfotos/resources/app/.webpack/renderer/main_window/index.html
scanning 1669846371858 2012
scanning 1670180191963 2005
scanning 1669820769237 2014
scanning 1670013768013 2007
scanning 1669977567062 2008
scanning 1669940026948 2010
scanning 1669794589742 2016
scanning 1669752053266 2018
scanning 1670038068463 2006
scanning 1669805667883 2015
scanning 1669853957429 2011
scanning 1669653259631 2021
scanning 1669833132983 2013
scanning 1669667329447 2020
scanning 1669971403943 2009
scanning 1669687698409 2019
scanning 1669631053429 2022
scanning 1670197124366 2004
Clusterization buffer read from cache successfully.

At this point Tonfotos stops occupying 25% of RAM, and when I close it that’s the output.

ApplicationClosed
Clusterization buffer saved, size= 34371456 faces: 56532
Clusterization buffer read from cache successfully.

(I disabled recognizing faces only after a lot of faces were recognized).

Andrey

Tommy I guess I know what is the reason. It’s faces clusterization, not scanning. Currently algorithm is that Tonfotos will re-run clusterization for every person in the archive, not only the ones who have been changed. However it does not do that more than once per day. And, if there were no changes in faces (no face was assigned to a person), then this process will not start at all. As I can see from your logs, clusterization buffer was in memory, that means program was working on clusterization during this run. I guess you have been manipulating with faces, hence the result, and that is why CPU was busy.

Tommy

Andrey I tried deleting the already scanned people. Nothing has changed, Tonfotos continues to bloat my CPU. This is the output in half an hour of open app. (Note how it cyclically scans the same folders).

$ tonfotos checkpoint config: 88 ms checkpoint app ready: 2544 ms [84744:1223/193411.150360:ERROR:sandbox_linux.cc(374)] InitializeSandbox() called with multiple threads in process gpu-process. Server running at http://127.0.0.1:9871/ scanning 1669820769237 2014 Auto update URL: https://tonfotos.com/distribution/tonfotos/release/linux/x64 intercepted file:///usr/lib/tonfotos/resources/app/.webpack/renderer/main_window/index.html MenuCommand group_remove_person intercepted file:///usr/lib/tonfotos/resources/app/.webpack/renderer/message_box/index.html Recreating clusterization buffer from BD. Allocated 34371456 MenuCommand group_remove_person intercepted file:///usr/lib/tonfotos/resources/app/.webpack/renderer/message_box/index.html MenuCommand group_remove_person intercepted file:///usr/lib/tonfotos/resources/app/.webpack/renderer/message_box/index.html scanning 1670013768013 2007 scanning 1669940026948 2010 scanning 1669846371858 2012 scanning 1669752053266 2018 scanning 1669653259631 2021 scanning 1670197124366 2004 Buffer creation time: 453714 faces: 56532 pos= 0 scanning 1669667329447 2020 scanning 1669833132983 2013 scanning 1670038068463 2006 scanning 1669805667883 2015 scanning 1669687698409 2019 scanning 1669977567062 2008 scanning 1669794589742 2016 scanning 1669631053429 2022 scanning 1669971403943 2009 scanning 1669756668306 2017 scanning 1670180191963 2005 scanning 1669853957429 2011 scanning 1670197124366 2004 scanning 1669971403943 2009 scanning 1669667329447 2020 scanning 1669833132983 2013 scanning 1669687698409 2019 scanning 1669756668306 2017 scanning 1669853957429 2011 scanning 1669971403943 2009 scanning 1669631053429 2022 scanning 1670180191963 2005 scanning 1670038068463 2006 scanning 1670197124366 2004 scanning 1669977567062 2008 scanning 1669805667883 2015 scanning 1669667329447 2020 scanning 1669833132983 2013 scanning 1669794589742 2016 scanning 1669971403943 2009 scanning 1669687698409 2019 scanning 1669853957429 2011 scanning 1669820769237 2014 scanning 1670197124366 2004 scanning 1669756668306 2017 scanning 1670013768013 2007 scanning 1669631053429 2022 scanning 1669940026948 2010 scanning 1669971403943 2009 scanning 1670038068463 2006 scanning 1670180191963 2005 scanning 1669667329447 2020 scanning 1669805667883 2015 scanning 1669846371858 2012 scanning 1669833132983 2013 scanning 1669853957429 2011 scanning 1669752053266 2018 scanning 1670197124366 2004 scanning 1669687698409 2019 scanning 1669971403943 2009 scanning 1669977567062 2008 scanning 1669653259631 2021 scanning 1669756668306 2017 scanning 1669794589742 2016 scanning 1669631053429 2022 scanning 1669820769237 2014 ApplicationClosed Clusterization buffer saved, size= 34371456 faces: 56532

Still thinking that disable scanning of old folders is a good idea!
(Picasa allowed this and I kept the scan always active only in the current year folder, the one where I uploaded new photos).

Andrey

Tommy I am still sure that CPU consumption has nothing with scanning, and most probably this is due to re-clusterization. To check this for sure I will provide you next week with beta version where this background clusterization will be turned off.

Tommy

Andrey ok, thanks!

mshen

more information for you…
as a new user of tonfotos, i had 10K+ photos for initial scanning and that took a while. During the 2 hours, my computer crashed twice (most likely due to overheating). I definitely attribute to the crash as an indirect result of the scan as this (older) laptop doesn’t crash often (maybe once a year, if that) and runs 24×7.

It’s a one time thing and subsequent loads of the application is not intrusive, so I don’t really have much concerns. Just another datapoint for you.