Michael K Johnson

It was interesting getting Mastodon up and running on AlmaLinux 9.

The CentOS 9 derivatives boot amazingly fast. At least half of my boot time to shell is waiting for grub! That's awesome for reducing the overhead of maintenance.

I ended up using Docker, which probably makes it take a few seconds longer to boot than if I just ran it all from source, but means I can manage all the services separately. It definitely makes tootctl take longer to run.

Jun 4, 2022 at 1:13pmWeb
Show threadMichael K JohnsonJun 4, 2022
I have raw notes of everything I did, which I do intend to edit into a blog post and publish. But probably half the wasted time was from not knowing that docker "internal" networks do not work with firewalld: https://github.com/firewalld/firewalld/issues/844
docker: firewalld 1.0.0-1.1 breaks inter-container networking · Issue #844 · firewalld/firewalld

What happened: Version 1.0.0-1.1 of firewalld breaks inter-container networking, at least for Mastodon's docker-compose.yml. Container mastodon_streaming_1 shows these errors repeatedly while tryin...

GitHub
Show threadDougJun 4, 2022
@mcdanlj interesting that Docker would have a firewall which mingles inside the instance networking. I can see it controlling container-2-container networking but not container internal networking.
Show threadMichael K JohnsonJun 4, 2022
@dougl It's intra-container (container-to-container) networking that is the issue — I'm running seven docker containers that need to talk to each other in certain ways, and while Docker now works with firewalld in general, that support doesn't extend to networks that are set as internal.
Show threadDougJun 4, 2022
@mcdanlj I would have thought it would boot faster since the kernel it uses is already booted. Unlike a VM where it's got to boot all from scratch. Either way, I enjoy reading your musings on this.
Show threadMichael K JohnsonJun 4, 2022
@dougl yeah, it's not booting the kernel that is slow. That's a second or two, then two or three seconds for the rest of the base OS to boot (mounting filesystems, setting up networking, etc). About 10 more seconds until the whole mastodon suite is started. It takes a while for the whole application stack to get to fully serving. One Postgresql, two redis (one persistent, one transient), one sidekiq, two ruby-on-rails apps. I haven't actually measured that but maybe another 30-45 seconds?
Show threadMichael K JohnsonJun 4, 2022

@dougl Ultimately, the site isn't responding for something like a minute after typing `shutdown -r now` — that's not terrible.

I could probably set up a "site maintenance page" like we have for Discourse that would temporarily show during the half a minute that mastodon is spinning up but not yet ready.

But I feel like I've done enough for now and can get to that later. Maybe after I write up my blog post. ☺

Show threadDougJun 15, 2022
@mcdanlj that's probably lots of disk I/O an the container. Makes me wonder if these containers have many layers to the storage system or if there's a way for the container to use host filesystem structure using host permission systems for access.
Show threadMichael K JohnsonJun 15, 2022
@dougl my recollection is that it's not IO. RoR apps boot slow. Ruby does not have an equivalent to Python's .pyc/.pyo files; unless something has changed since I learned about this, it's not ever compiled. Most folks don't seem to understand that Python is a compiled language with a compiler in the runtime. Ruby is purely interpreted as far as I know.
Show threadMichael K JohnsonJun 15, 2022
@dougl the shared volumes are exactly what you say, and that's where all the files that persist between updates live. Database, media, logs, etc. The containers use an overlay file system such that content between related containers is literally backed by the same physical memory pages in page cache, and faulting those pages into the container process is just mapping an existing page for identical content for the hot cache case. It's actually reasonably parsimonious.