Building a #Linuxulator userland from source on #FreeBSD -- I'm not giving up yet. But, starting over π
This time, I try building some "complete" #GNU cross-toolchain targeting #Linux first in some dedicated prefix. Maybe this will finally work out. π
Progress again. 
GCC builds fine when I previously install binutils in the exactly *same* prefix!
Ok, this means I'll need an extra "bootstrapping" port for binutils as well...
Time for bed!
Been a week now I'm working on this idea ...
So far, whenever something looked like good progress, I ran into some dead end one or two ports later.
Should probably decide for some #timebox I will set myself for this. If I can't at least come up with a *working* cross (#FreeBSD -> #Linux) #GNU toolchain within that timebox, the idea should be abandoned as unfeasible π
I'm carefully optimistic now again π
After first building very basic/limited "-bootstrap" versions of binutils and gcc into a separate prefix, it seems I could finally build a complete #GNU cross (#FreeBSD -> #Linux) toolchain, including #binutils, #glibc and #gcc (with libstdc++). This final cross gcc at least passed the most basic sanity check -- it successfully compiles an empty program π
Now doing a bit of cleanup and then trying whether this beast is able to build the *real* (native) glibc for a new #Linuxulator userland π
And we're getting closer ... this new cross-toolchain successfully builds a "native" glibc as the first package for the new #Linuxulator userland π₯³
So now, just a few more ports until I reach one that will require C++11 threading functionality ... that's where it finally broke beyond repair with my first more naive approach (without any "bootstrapping" ports/packages).
Ok. #zstd was the first #Linux port that needs C++11 threads. And, it builds!
My #GNU cross toolchain on #FreeBSD finally seems to work! π₯³
Time to party, hehe. Or, time for bed in fact.
Next step will be a *native* Linux toolchain to build the rest of the #Linuxulator userland.
https://github.com/Zirias/zfbsd-ports/blob/linux/archivers/linux-zstd/Makefile
This feature-branch finally starts to look somewhat promising, just added native #Linux #GNU #binutils for #FreeBSD's #Linuxulator π
linux_base-dirs is a tiny metaport just owning ${LINUXBASE} and a few essential subdirs.
Everything prefixed "lxcross-" is part of the cross-toolchain and installed to ${LXCROSSBASE}, defaulting to ${LOCALBASE}/linux-cross.
Everything suffixed "-bootstrap" is some minimal/temporary port needed to build the full cross-toolchain and installed to ${LXBOOTSTRAP}, defaulting to ${LXCROSSBASE}/bootstrap.
Everything prefixed "linux-" is Linux-native and installed to ${LINUXBASE}.
linuxheaders44 just contains the headers from Linux-4.4.x, installed to ${LINUXBASE} but coming with a slave-port installing to ${LXCROSSBASE} for the cross toolchain.
Next step: GNU gmp! Let's see π
We have #glibc, #zlib, #binutils, #gmp, #mpfr and #mpc ... in theory everything needed to build a full-featured native #gcc for C and C++. Oh wow. Now, trying to create *this* port π
Edit: My hope is that with the --sysroot option (set to ${LINUXBASE}), this new toolchain will only ever look for libraries inside ${LINUXBASE}, avoiding weird build issues you might get when using the existing linux-c7-devtools port. Well, I'm not sure I fully understand this --sysroot magic π
Another little update on this endeavor .. native #Linux #GCC for #FreeBSD's #Linuxulator finally builds! π₯³
Still needs some "love" in detail (like, correct path for man files).
And also, I have to find a way to package #libgcc separately, cause most software built for Linux using GCC will need it, and it doesn't make sense that everything pulls in the whole GCC as a dependency ...
Well, maybe tomorrow then π
Indeed, sanity-checking the new #Linux-native #GCC for #FreeBSD's #Linuxulator shows it works perfectly fine at least for C, and using static #libgcc (because I removed the libgcc_s.so file, should really be packaged separately) π₯³
So, to complete the toolchain, next steps will be separate ports for libgcc and libstdc++, should be possible to build these with this native gcc, we will see!
He, finally. The #Linux-native #GNU toolchain for #FreeBSD's #Linuxulator just passed a first sanity check using C++ π₯³
I guess this means I could start creating ports for a basic userland now... π
Trying to get this to the other archs supported by #FreeBSD's #Linuxulator (amd64 and i386), I'm hitting yet another wall: #Linux headers fail to build there. Oh freakin hell, I expected to do quite some adaptions to support all 3 archs, but I didn't expect it to fail at *this* stage π€―
Already tried to understand what happens in Linux' build system. Well, it's not exactly easy to understand π
Shot in the dark, trying it with #gcc (instead of the base #clang) now... (which will take a while cause my test machine has to *build* this gcc first, bah ....)
Well ... no, building with #gcc didn't help.
Turns out the issue were #ELF relocation types. I have no idea why #Linux needs them *just* to build the headers (which is all I need for #Linuxulator, cause the actual #kernel will *still* be #FreeBSD), but yep, one type is named slightly differently: R_X86_64_JUMP_SLOT on Linux, R_X86_64_JMP_SLOT on FreeBSD. It has the same value, 7, so for now I just hardcoded it in the port Makefile for the Linux headers. π (hey, it works π€·ββοΈ)
@thindil No, that's not the issue, it was really just about this slightly different spelled constant for some ELF relocation type ... it works now.
I'm btw only testing on 14 so far. But I use #Linux 4.4.x for the headers (to be compatible with FreeBSD 13). I will add a metaport later that will pick Linux 5.10(?) on 14 instead.
For the #GNU toolchain, I'm using the newest versions in any case. They're still compatible with Linux 4.4 π
@thindil Thanks!
I'm almost done with at least the cross (#FreeBSD -> #Linux) #GNU toolchain for all supported archs (aarch64. amd64 and i386), currently working on #glibc for it. It's a bit tedious work because all parts of the toolchain seem to have a pkg-plist dependent on the target architecture π
@thindil That's what I assume as well, once I'm done with the toolchain, architecture-specific differences should be rare (if encountered at all). π
Just finished the cross-version of #glibc for all archs. Two more (#binutils and #gcc) to go for the FULL cross-toolchain. That will be the goal for today, the native toolchain maybe tomorrow π
@thindil it sure does. Another one of these things that *should* be solved once the *native* toolchain is completed.
Just redefine ${STRIP_CMD} (for direct usage), and when used from "install -s", it should then automatically use the "native" strip, because the toolchain is compiled with --with-sysroot=${LINUXBASE} ... we will see!
@bdiederik No. #Debian's #kfreebsd is a #GNU userland configured to use the (native) #FreeBSD kernel. What I'm building here is a #GNU userland using a #Linux kernel (although "faked" by #FreeBSD's #Linuxulator -- the kernel can provide a Linux-compatible userspace-ABI, syscalls etc...)
The goal is to replace the ancient "linux-c7" userland currently offered in FreeBSD ports. I'm testing here whether building a userland from source would be a feasible way π
@meka @bdiederik Right now, all I have is a working native toolchain. On aarch64. Next step, test on i386 and amd64 (to cover all archs supported by #FreeBSD's #Linuxulator at all) and do the necessary adaptions.
Once *this* is done, I can try to start building an *actual* Linuxulator userland from source.
I'm certainly not talking about documentation at this stage π
But thanks, nice to see people are interested in this project. If it works out at all (still not sure), I *am* sure I won't be able to maintain it all alone π
@zirias A more functional and easy to use #Linuxulator would be awesome!
@hnygd @meka @bdiederik The "normal" approach would be to find another (newer) Linux distribution and re-package *that* instead of Centos-7.
And maybe, this is the better approach, I'm not sure yet. But I really want to try the "build it all from source" approach as well π
@jhx It would be a lot less frustrating if I wouldn't have to sit and wait all the time (e.g. rebuilding my minimal cross-gcc takes around 11 minutes).
I wonder whether #ccache with #poudriere would also work for some custom-built cross-compiler? π€
@zirias
True, the waiting is what makes it annoying!
Mhh, not sure about that... in general: sure, you could leverage ccache... question is how good it would work.
I know that ccache can sometimes bork builds
@zirias Just a stupid question here. π What exactly is the purpose of the project? Because if it is just cross-compiler from #FreeBSD to #Linux, you can use:
1. Linuxator and GCC there. That could be literally native compilation.
2. Get the proper libraries and use clang cross-compilation abilities. That solution is more native for FreeBSD and require less disk space.
@thindil Background is that #FreeBSD's #Linuxulator userland is in desperate need of an upgrade. The "linux-c7" userland is both too antiquated and incomplete for a lot of #Linux software you might want to run on FreeBSD.
Purpose of my project is to evaluate an approach of building a new userland from source instead of repackaging another Linux #distribution. The rough plan is to create a feature-complete native #GNU toolchain for Linux and then use that to build the Linuxulator userland packages.
The benefit I would *hope* for is that this could allow to relatively quickly create new packages for the Linux userland from existing FreeBSD ports (maybe using "slave ports"), but this has to be proven in practice.
@thindil I'm actually using #LFS as a reference to get around the weirdest issues "bootstrapping" some #GNU toolchain at all. But of course it can't be used as is, LFS expects to be built on some Linux system and even uses #chroot to build some packages, definitely impossible using #FreeBSD #ports.
I certainly don't want two package managers on my system. Plus, a #Linuxulator userland should only install what's *different* from the normal FreeBSD userland, because /compat/linux works as an "overlay".
Trying to create new ports/packages for the Linuxulator userland is cumbersome, first you have to rely on what the repackaged distro offers, second you have to understand the special repackaging magic used.... that's exactly what I *hope* to improve by just building from source. No, I'm not convinced myself yet this is a good idea, I have to try!
@zirias True, the only way to know do it's a good idea is to test it. π And before I forgot again, there is also Chimera Linux: https://chimera-linux.org/ Maybe it will be helpful.
Linuxator isn't exactly an "overlay". #FreeBSD has implemented two APIs in the kernel and can detect, on a binary execution, via magic bits, which API to use. Thus you can't mix, for example FreeBSD library and Linux binary. We need to deliver the whole dependencies chain for that binary.
@thindil I can certainly use #FreeBSD "tools" (gmake, bison, gettext, whatever) for building this cross-toolchain. But indeed, for libraries, they need to be built targeting #Linux. And because #GCC with the full feature set needs e.g.#glibc when targeting Linux, but then you need GCC to *build* glibc, I need at least some "bootstrapping" ports. It's really a mess.
Once I have a full-featured cross GCC targeting Linux ready, I'll stop for a while to party π
@thindil Oh there *is* binary compatibility for sure. The #Linux kernel typically doesn't break its userspace-facing #ABI. #Glibc and #GCC's libstdc++ use symbol versioning to provide backwards compatibility.
The issue starts with all the other libs, there's no standard for some "base" GNU/Linux system. That's where all these (IMHO damn broken) ideas like #AppImage, #Flatpak etc come from. Of course, you could just link statically instead, seems people don't get that any more π
Anyways, quite some binary #Linux software will work "anywhere" as long as the required libs are not too old (looking e.g. at browsers...). And having a #Linuxulator userland built from source *should* enable you to just add ports for missing libraries. Well, in theory π
@zirias 1. From my experience, breaks. Reason: distros use different build flags. And whole piles of patches. π
2. Unless there's some security problems, then they break backward compatibility too. π€£
Btw, I'm sure that I forgot to mention that there's no standard C library in Linux world? IIRC, there are 3 or 4 incompatible ones. The two major are GNU lib C and musl. π€― Ubuntu ships them both. π
@thindil
1. This doesn't affect the public #ABI of the kernel. Do you have any example of breakage here? π€ Don't get me wrong, I'm really not a fan of the chaos GNU/Linux actually is, but this is still hard to believe.
2. Possible. I don't know any example though ....
Regarding the "standard" #libc, if we're talking about "GNU/Linux", this will always be #glibc. AFAIK, any #Linuxulator userland ever shipped in #FreeBSD ports used it. I plan to do the same trying to build this mess from source π
BTW, my "lxcross-glibc" package seems to work, and it seems I'm able to build some *almost* complete cross #GCC using it right now ... still need the --disable-libsanitizer configure flag π€
@zirias 1. Mostly from work. An old binary was depending on some old, insecure syscalls which were disabled by default in later versions of the kernel. Sure, rebuild kernel was fixed the problem. But it's hard to call that: "we don't break API". π Sometimes I have feeling, that FreeBSD has better compatibility with Linux than all these distros.
And nice result, congratulations. π₯³ Let's hope the best with the rest. π
Felix Palmen π―
Bartek Jasicki
Bram Diederik aka daft_dutch
meka
Stephan Lichtenauer | Χ Χ Χ‘ΧͺΧ
jhx
probono
Michael Dexter