| Code | https://github.com/rburchell/ |
multibuild v0.0.2 is out.
New feature: output formats: https://github.com/rburchell/multibuild?tab=readme-ov-file#output-formats
So if you want to distribute tar.gz or zip, as well as, or instead of your #Golang binaries, that's now easily possible:
//go:multibuild:format=raw,zip,tar.gz
[master 0fae0d8b] pkg/irc: who cares?
6 files changed, 970 deletions(-)
RIP to some of the first #Golang I ever wrote. I don't think I've used it since 2021, and. the last *serious* use was probably 2014.
Hm, one of the few times I've noticed a missing piece in the #Golang stdlib: compress/bzip2 only has a decoder, no encoder. That's frustrating.
I wanted it for output formats for multibuild: https://github.com/rburchell/multibuild/pull/16
Seems it's quite an old issue:
https://github.com/golang/go/issues/4828
It blows my mind that #Github Actions doesn't have an officially supported action for creating releases.
Yes, I'm aware community actions for that exist, but I don't want to take unnecessary supply chain security risks.
I finally cracked the other day. Yet another small Go binary that I needed to test on a few different machines with different GOOS/GOARCH.
Meet multibuild: https://github.com/rburchell/multibuild
It's a simple way to make cross compilation of #Golang binaries easier. One command to build for many platforms.
And yes, I know that there are other options. But this one's mine.
Found another person to have gone down this rabbit hole here: https://blog.arkey.fr/drafts/2021/01/22/native-memory-fragmentation-with-glibc/
To conclude this very long thread, what I'm wondering is twofold:
1) isn't this many-arenas-per-core approach going to generally make life miserable on devices with a lot of cores but low RAM?
2) I wonder how much of the blame glibc's fragmentation handling gets is really just problem 1...
Some research tells me that jemalloc also has per-thread arenas, but it looks like it uses a lower (4*core) number...
... anyway, at least on paper, I'm not supposed to be working now ...
5/5 (END)
The app in question is a heavy user of short-lived threads. Fragmentation is also an issue as we have a fair bit of allocs of differing sizes/lifetimes (typical C++) - some allocations are passed between threads and live a long time etc.
So, my theory is each of those arenas separately ends up fragmented, and things get worse until we run low on RAM.
Doing some reading around (e.g. search for MALLOC_ARENA_MAX) shows that this isn't _really_ news..
As an example: https://sourceware.org/legacy-ml/glibc-bugs/2010-02/msg00050.html
4/5
Specifically, this bit:
> The default value of this tunable is 0, meaning that the limit on the number of arenas is determined by the number of CPU cores online. For 32-bit systems the limit is twice the number of cores online and on 64-bit systems, it is 8 times the number of cores online.
Aha.. Modern glibc malloc uses arenas that are assigned at first allocation per thread up to a max, when it then starts reusing them!
3/5
