Turning One

by @cambraca

Musicians Today has officially completed its first trip around the Sun, and I'd like to celebrate the fact that we're still here. Existing is a wonderful thing.

We were born in the middle of the Twitter chaos with the goal of providing a space for people who are also musicians, who want to get away from traditional “you're-the-product”-style social media. It's an experiment, as these things usually are, but a successful one if I say so myself.

The #MastoAdmin of it all

But it's also a time investment.

Maintaining a Mastodon server is such a balancing act if you're not willing to throw all the monies at the problem. You have your disk space, RAM, and CPU usage to worry about, and then your multiple processes running on different technologies (Java being my least favorite—why would you greedily consume half of the available RAM, however large it is, as your default behavior?), each with its own flavor of configuration format. You have all kinds of monitoring software (including the one I created myself for the Status page). You have costs that can vary out of the blue (yes, I'm running it mostly on AWS which is not cheap, precisely, though in my defense it is pretty reliable). You have to have moderators, of which I have two for whom I'm neverendingly grateful because they're awesome. And, last but most-definitely-not-least, you have the (endless but expected, but endless) drama that stems from all human communities, which in Mastodon/Fediverse-land seems to take the form of (de)federation wars.

Back on the technical side though, here's an incomplete list of stuff I've learned to monitor/configure/parent:

• The three main Mastodon processes (web, sidekiq, and streaming). Each one is its own Very Important Piece of the experience, can fail in different ways, and can also be optimized in different ways (e.g. creating more than one process for Sidekiq for parallelization).
• So many Linux commands for keeping track of: free disk space but also inodes; memory usage, both real and virtual; I/O stats, because the disk sometimes can't keep up; the maximum number of files open per user (don't ask); and logs, oh so many logs for each different service.
• The file storage service, because Amazon S3 is way too damn expensive so we're using Backblaze, which is cheap, but since it doesn't completely implement the S3 API, Mastodon can't, for example, set permissions on individual files.
• The services adjacent to Mastodon: Postgres (pretty important, you could say, since it's the main database) but also Redis (which is surprisingly critical in Mastodon and not just a caching layer); Elasticsearch (only if you want search, but you do); Prometheus and StatsD and the custom thing someone built to convert Mastodon stats to a format that Grafana can understand (only if you want metrics, but you do); Cloudwatch (Amazon-specific, yes, but it's what tells me when the disk is running out so... pretty essential).
• The random dependencies: ffmpeg (which tends to murder even the toughest server if folks start uploading too many videos); imagemagick (which, if you're on an old version, images can suddenly stop showing altogether).
• The custom code that needs to be merged every time there's a new Mastodon release, because why do this at all if you can't personalize things a bit?
• The Patreon page cuz no money, no infra.

I guess it's fitting that we had our first real Sidekiq emergency right before the anniversary—a rite of passage, you could say.

Thank you

But it's worth it. It's worth it because, from what I can tell, at least some people are getting something good out of it. I'm a part of that group, by the way.

I guess what I’m trying to say is thank you. To the ones who get actively involved. To the ones who stick around for any period of time. To anyone who tries it and moves on, too. You're all appreciated. Let’s keep it up!

💬 Feedback/comments