last modified: Sun Jan 09 2022 15:24:21 GMT+0000 (Greenwich Mean Time)

* using s3fs for navidrome -- learn from my mistake!

using s3fs for navidrome -- learn from my mistake!

you may be like me and enjoy self-hosting stuff instead of paying someone else subscription fees to rent your media, which can be taken from you at any time, and may not even be an exhaustive collection of everything you want to listen to. i've been managing a local music library for years with FLACs and MP3s and more obscure formats as well.

i recently bought a new phone, a used google pixel 3a which is now running CalyxOS. i wanted my music library on there, unfortunately my music library is over the 64GB of storage the phone has built-in, and i didn't want to manage going through, selecting different albums to store locally on my phone all the time (though it may be a good idea to store some in case there's no mobile internet). i have recently also set up a back-up system with backblaze B2 and kopia (post maybe coming soon?) so i thought it was a good idea to put my music library on B2 and then use navidrome on my server to access those files, and serve them to me when i was away from my main computer.

** first problem

i found out that navidrome currently doesn't have support for S3-compatible storage and i didn't want to use much of my server's local block storage as that i thought that would limit me too much. so i turned to s3fs-fuse, a POSIX compatible file system layer where you can use many standard linux file system functions, i.e. listing, creating, deleting, moving files, but it all resides in S3-compatible storage, so i could use my B2 storage and not use up much of my server's local storage.

** success?

after setting up s3fs, configuring navidrome and s3fs systemd services and copying my music library via rclone (this one takes a while), creating a subdomain, nginx reverse proxy and ssl certificates, everything seemed to have worked. i installed the wonderful subtracks mobile app and everything worked as it should. relieved to be able to move on from the project, i went to do other things for a few weeks....

** here's where the screw up comes in

i went to check my backblaze B2 account out of curiosity, making sure everything is working as intended. i was slightly stunned looking at the "billing" section on my account page. that was an excessive amount of money paid for this amount of storage... what on earth is going on?!

backblaze b2 billing page, showing the amount of money i paid for my ridiculous amount of class C calls

my class C transactions cost over $40, now that seemed a bit ridiculous. i checked my "reports" page to see what class C calls were causing this:

backblaze b2 report page, showing the number of s3 list object calls
that is a big number...

over 3 million S3 list object calls. now that was seriously impressive, in a not very good way. s3fs was the only program i was using that was using these calls to my knowledge, so i knew something was up with my navidrome instance. navidrome maintains a database, so we shouldn't have to use a list command that often, except for scans of the music library. navidrome comes with a scan scheduler, where i expected the scan schedule to be of some sort of modest value, maybe once a day or so.

the configuration page in the navidrome documentation had the default value at "@every 1m", i.e. every minute navidrome was running, it would perform a scan of the music library. so uh yeah. that ended up costing me more money than i wanted to. but hey i'm glad it didn't go into the hundreds or more! keep and eye on your running services people, alerts and caps are the smart way to go!

** the fix (and a few other miscellaneous changes)

a few days after panic stopping the navidrome instance (i.e. today), i set ScanSchedule = "0" in the configuration file. it seems to be running well now, without making tons and tons of these list calls.

another change i made a while ago was setting Scanner.Extractor = "ffmpeg" in the configuration file. some of my music wasn't well formatted metadata-wise (even some albums i would buy, there's no excuse) and this allowed them to be played without navidrome complaining. also just today i figured out transcoding (you enable it per player), whereby i'm able to stream s3xmodit module files (i.e. .s3m, .xm, .mod, .it) which i have a fair amount of as part of my collection.

so yeah. another expensive mistake in the bag. whoops.


set ScanSchedule = "0" in your navidrome.toml configuration file. otherwise, it may cost you a pretty penny.

Creative Commons License
All text related to this post is licenced under a Creative Commons Attribution-ShareAlike 4.0 International Licence unless otherwise stated. Images or other media may be under a different licence if stated or may have all rights reserved.