Mastodawn

AndresFreundTec

Postgres until now has been using cirrus-ci for CI. Unfortunately they're shutting down June 1st.

They had been a good fit for us because:
- some free credits so everyone could run some CI on their own
- supported providing own compute for more demanding cases (via GCP and self hosted mac HW)
- jobs ran in one-off VMs
- support for custom images allowed us to pre-install everything, keeping test times manageable
- VMs not containers was good for perf
- CI job definitions could be run locally

AndresFreundTec 3d ago

Any suggestions / experiences where to look next?

AndresFreundTec 3d ago

For our own compute we've been averaging daily:
- 1464 core hours (full cores, not SMT)
- 396 of which were windows (visible due to the licensing cost)
- 40GB of artifacts
- doesn't include macos, which I can't track as easily, due to being self hosted runners

So we will likely need something where we can continue to provide compute ourselves, to keep this affordable.

Vincent Garrigues 2d ago

@AndresFreundTec buildkite is great

@AndresFreundTec Laminar CI is great, you can build your own customized advanced CI on top of it. E.g. Vagrant with VitualBox, or KVM. Or anything else really.

Schreini 3d ago

@AndresFreundTec

@WoodpeckerCI might be an option

@Cs137 @AndresFreundTec @WoodpeckerCI

seems like codeberg may be a good place to go, since they'd host you free, being an opensource project, and they use woodpecker

https://docs.codeberg.org/ci/

Working with Codeberg's CI | Codeberg Documentation

AndresFreundTec 3d ago

@Viss @Cs137 @WoodpeckerCI
I suspect our CI usage would very quickly make codeberg not like us, it's probably too much given their size.

Looks like woodpecker doesn't quite have the support for running full VMs, but I guess they do have a plugin architecture for that...

@AndresFreundTec @Cs137 @WoodpeckerCI it might be worth investigating fly.io, i think one of their leads is here on mastodon too - they might have some kinda 'if youre a big opensource project we can give you huge discounts and/or credits' deals

Max Nordlund 1d ago

@AndresFreundTec @Viss @Cs137 @WoodpeckerCI @Codeberg

I'm pretty sure both Woodpecker and Forgejo Actions allow for self-hosted runners. That would shift the cost from codeberg to you, but it sounds like that wouldn't be a problem.

Woodpecker CI 1d ago

@AndresFreundTec @Viss @Cs137

We currently have no native VM support, but this is only because of some open implementation questions: https://github.com/woodpecker-ci/woodpecker/discussions/2302#discussioncomment-6775117

feedback is welcome!

And writing a own backennd is easy and transparent to the server (so you could use #codeberg 's hosted CI and use own agents with custom backend ;) )

(You just have to implement this interface: https://github.com/woodpecker-ci/woodpecker/blob/main/pipeline/backend/types/backend.go)

Libvirt backend support · woodpecker-ci woodpecker · Discussion #2302

Clear and concise description of the problem Docker containers offer a majority of the cases needed for builds, but in the case of non-Linux systems like OpenBSD, NetBSD, etc., launching complete v...

GitHub

DeepBlue V7.X 3d ago

@AndresFreundTec We are using tart with gitlab-ci. It is not clear yet, if that will also get shut down in the near future, since that was their VM software, but at least for now it seems to still be maintained, so maybe that would be an option? For Windows we just use gitlab-ci with a custom runner, that starts libvirt VMs. It is quite flexible and probably could be made to fit most things, but it does take some effort to setup and maintenance as well as compute hardware.

However we already run our own Gitlab as well, so it is quite likely, that our solution is not a good fit at all for you. But I do think running your own CI hardware using open-source tools wasn't a bad idea.

Vadim Rutkovsky 2d ago

@AndresFreundTec buildkite with selfhosted runners could be an option

@AndresFreundTec talk to Taylor at https://pixelair.io/

Pixel Air IO - Concourse Experts

Pixel Air IO provides commercial support for Concourse CI from the maintainers of Concourse CI

Renaud Chaput 2d ago

@AndresFreundTec I would look at what curl is doing, and even ask @bagder

daniel:// stenberg://2d ago

@renchap @AndresFreundTec the #curl project is fortunate to have GitHub as a sponsor so we get a lot of CI CPU in their infra for no money == we run most of our CI on GitHub

@bagder @renchap @AndresFreundTec doesn't GitHub offer unlimited CI usage for public repositories anyway? At least it did last time I checked a few months ago

daniel:// stenberg://2d ago

@grishka @renchap @AndresFreundTec I think they mostly cap the parallelism

Shlee fooled around &2d ago

@AndresFreundTec I'm somebody who recently moved to self hosted in colocation... depending on the budget it's easy and "cheap compared to cloud services" (getting second hand servers isn't a huge deal)

I live in Australia so the costs are 3-10X higher than the US/EU... but I'd imagine you could get a bunch of open source projects to go into building a CI Farm or just build it for yourself

I assume you were getting a good deal before but right now, I advocate for moving off the cloud when the price is right.