Mastodawn

✧✦Catherine✦✧

I really hate dnsmasq. every once in a while I have a problem and some Linux guy tells me "just use PXE bro" and then I spend a hour configuring PXE that doesn't fucking work without displaying a single debug print and all of this is a complete waste of time.

the UI of dnsmasq is atrocious; if I wanted to be able to tweak every DHCP option I would use my TCP/IP stack to answer DHCP queries. just give me something that can boot a machine without four hours of twiddling with options that are required for correct operation yet nobody bothers to give them names better than "66"

✧✦Catherine✦✧Feb 27

I don't think there's been a single time in my life when I successfully PXE-booted something outside of a perfectly-controlled, utility-free environment with a crossover Ethernet cable

✧✦Catherine✦✧Feb 27

this time I can't even get it to boot in a completely artificial environment. the PXE boot ROM doesn't log anything to the display and when it errors out it shows the error for like 100 milliseconds before switching to the next boot option

fucking useless technology

✧✦Catherine✦✧Feb 27

anyway if you ever recommend anybody use PXE for anything who isn't being paid a salary or a consulting rate for setting this up: don't.

Manawyrm | Sarah Feb 27

@whitequark as someone who's paid a salary to do this: it's just as awful. It really is this buggy and you constantly fight against insane software/firmware bugs...

✧✦Catherine✦✧Feb 27

@manawyrm but at least you're being paid for it!

Manawyrm | Sarah Feb 27

@whitequark but if you are required to get it to work and giving up is not an option and you'll then have to tell ASRock, ASUS, Dell, etc.
"Hey, trashing the stack when returning from a function call really isn't cool and a bug, pls fix" and they look at you like???
you really are in a bad situtation and well, it's frustrating :P

If you ever do want to try again:
tftpd-hpa and iPXE as a second stage loader work well. OpenWRT as a configurable DHCP server also works well.

@manawyrm @whitequark it works as long as the vendor doesn’t fuck up the MSI-X / IRQ interrupts or uses an ancient EDK2 base

Manawyrm | Sarah Feb 28

@wyldtom @whitequark … or the customers want legacy boot or you need IPv6 or or or or or….

@manawyrm @whitequark Ah yes: Once had a server that for some reason appended a 0xff byte to the pxelinux filename it got via DHCP. Solved it by renaming the file on the TFTP server to include the 0xff byte at the end 🫠

Samantaz Fox Feb 27

@manawyrm @whitequark Reminds me that one day where I tried to boot a laptop from PXE, but it couldn't connect to the server. After rmuch digging around and wiresharking, I found out that the DHCP client in the PXE ROM had an "off by one" bug, that caused it to read an extra nul character from the "next server" address, thus trying to resolve an invalid domain. Fun times!

Manawyrm | Sarah Feb 27

@SamantazFox @whitequark *sigh*

@whitequark did you try UEFI HTTP boot? ( https://www.ytvwld.de/blog/netboot.html ) In my experience this is harder to setup on the client but way easier to setup on the server.

How (not) to netboot

HTTP is cool – if it works

ytvwld.de

✧✦Catherine✦✧Feb 27

@ytvwld no... because I've no storage on this machine on which to put the ESP. but I might have to solve that first

@whitequark depending on the firmware you might not need to, just putting a url in a boot option might be enough if your system is new enough

✧✦Catherine✦✧Feb 27

@ytvwld it's a xeon e5-2630v4

@whitequark IIRC I once tried PXEbooting the Debian installer (it's just a DHCP option and some files on TFTP, how hard could it be?), gave up, and burned the CD instead.

(Lots of single-use CDs were burned at that job, I was trying to reduce that slightly. I guess these days you'd stick it on a flash drive which is a lot less wasteful.)

I love this, so I Feb 27

@whitequark hard agree, as someone who routinely sets up various network boot environments for cash money.

It’s the Linux of network booting technology - whoever designed PXE looked at everything else in the market and decided to try being everything to everyone instead of just doing one thing simply using standardised configuration options.

Hugo Mills Feb 27

@whitequark At one point, I did have PXE working on my home network. I even used it regularly for booting my diskless media player (with NFS root). I also had it set up with Debian install media. So I can confirm that it _can_ be made to work, but I'd think twice about setting it up again.

I probably still have the DNS/DHCP entries set up to point at a non-existent TFTP service (which has caused me trouble more recently).

Dimly Lit Corners Feb 27

@whitequark Can't it log via #Syslog to another device on lan?

(I'm a complete PXE, NetBoot noob but interested )

✧✦Catherine✦✧Feb 27

@DLC it's not Linux. it's what some crackhead shit the vendor has put into the UEFI with no explanation or much testing

Dimly Lit Corners Feb 27

@whitequark v.v

Our sincere condolences

#UEFIConsideredHarmful

✧✦Catherine✦✧Feb 27

@DLC I misremembered, actually. this particular flavor of crackhead shit predates UEFI (it requires CSM to work) and lives in the network card's option ROM.

Will Glynn Feb 27

@whitequark Can you replace the terrible NIC boot firmware with an iPXE build you control?

https://ipxe.org/howto/romburning

(Edit: presumably `make bin-x86_64-efi/[PCI ID].efirom`, per https://ipxe.org/appnote/buildtargets)

iPXE - open source boot firmware [howto:romburning]

✧✦Catherine✦✧Feb 27

@willglynn yes actually; it would be somewhat of a pain to actually do it (not sure if the flash is in-circuit-programmable and my guess is no)

Will Glynn Feb 27

@whitequark It has been Linux-flashable memory on all the NICs I've tried ¯\_(ツ)_/¯

iPXE gets you the ability to embed scripts as well as lots of optional extra functionality. More than once I've shipped a box with it configured to netboot by chainloading to a script in cloud object storage, at times even using static IPs with static VLANs. In your case, you might like syslog 🙂

The easiest way to test any of this is to build iPXE USB sticks, or if you have a BMC, iPXE floppy/CD images. https://netboot.xyz uses these targets (and embedded scripts) to netboot directly to their CDN, relying on DHCP for connectivity but not for boot directives.

Your favorite operating systems in one place! | netboot.xyz

netboot.xyz enables you to PXE boot many Operating System installers and utilities from a simple to use menu powered by the iPXE project.

✧✦Catherine✦✧Feb 27

@willglynn I am quite sure it's Linux-flashable but my problem up to this point was that I can't get any Linux image I've tried to boot (I have since done it which means I no longer have to fuss with PXE)

Chris Petrilli Feb 27

@whitequark I did it once.

But I forgot how.

Chris Petrilli Feb 27

@whitequark but seriously, though, it is doable, but man is it not fun.

Twitter's edge was PXE booted because everything that potentially handled clear-text traffic was not allowed to have any durable storage.

You get PXE and you get a TPM.

Chris Petrilli Feb 27

@whitequark while it was pretty stable, it didn't get there easily, and it still had some very... "curious" choices.

✧✦Catherine✦✧Feb 27

@petrillic yeah, I don't doubt that it is doable when you are on a salary for an enterprise deployment. when you have one (1) machine whose UEFI vendor you cannot go and berate until they fix things if something goes wrong, it is completely different

Chris Petrilli Feb 27

@whitequark absolutely. the chances of me ever bothering to try and do it at home are ... vanishingly small.

Even though the idea of a PoE powered PXE booting device is SUPER interesting.

@petrillic @whitequark does the firmware flash chip not count as durable storage?

Chris Petrilli Feb 27

@mei @whitequark let us hope that nobody is storing cleartext from TLS sessions in the firmware storage.

But certainly, yes, theoretically, it does.

@petrillic @whitequark oh right im dumb forgot about what the threat model is here

new question: does the database server not "potentially handle cleartext traffic"?

Chris Petrilli Feb 27

@mei @whitequark so I left out lots of details for varous reasons, but this was specifically about systems that were not under full physical control of the company. While we ran our own datacenters, we also had points of presence (POP) all over the world, often in colocation facilities where the security might not be quite up to our expectations.

It also helped deal with potential government seizure issues since we wouldn't likely get any notification from the colo provider, versus the experience in our own datacenters.

The systems (called TSA, Twitter Streaming Aggregator, if I recall the meaning) were the first line of unwrapping traffic for routing. It then got stuffed in an mTLS session to systems that were inside our datacenter.

So, keeping those keys as safe as possible was paramount.

David Chisnall (*Now with 50% more sarcasm!*)Feb 27

It wasn't PXE, but we used BOOTP to boot a room full of SPARCstation 2s from a single Pentium server. They ran an X server and used the Pentium to run apps as well.

✧✦Catherine✦✧Feb 27

@david_chisnall yeah, I've also seen enterprises where this works (including I think my high school?)

just never managed it myself

@whitequark theforeman automates pxe booting config (there’s probably other projects now too), but now you have two problems — theforeman and PXE.

✧✦Catherine✦✧Feb 27

@wbftw this is not an improvement.

@whitequark like you alluded to elsewhere in the thread — PXE is a PITA and starts to make sense in larger environments.

algernon.st Feb 27

@whitequark the only time i have used pxe boot (and have it working), was in an azure VM that used SCCM to configure the PXE booting for a WIM file. might as well been fucking magic since it didnt require much of my work beyond rooting around an ISO to get anything working.

Ted Mielczarek Mar 2

@whitequark years ago at Mozilla I got PXE booting working for the TI Pandaboards we were putting in custom rackmount enclosures to use for Android/Firefox OS testing. It was pretty annoying.

Kurt Kremitzki Feb 27

@whitequark It's wild that the usability of dnsmasq made me appreciate the interface Windows Server has for DHCP options :(

@whitequark I sysadmin this professionally (not dnsmasq though, I've always found that to be a rather awful experience) and have even managed to set it up at home once or twice, but that experience left me reaching for most other tools to boot a system before setting it up again.

Once it is up and working etc, it's great. The way there is awful. And I even know what tools to reach for from a couple of decades of professional experience.

✧✦Catherine✦✧Feb 27

@maswan that's validating at least ^ thanks

Jernej Simončič �Feb 27

@whitequark I usually go for for TinyPXE on Windows, simple to set up and works well enough for the stuff I need it for.

(I did once set up Windows PXE boot with WDS, but that's quite a bit more involved; was worth it though, because we had to image 170 tablets).

TinyPXE Server – Erwan's Blog

Samantaz Fox Feb 27

@whitequark Yeah, on my server I replaced dnsmasq with isc-dhcp. The latter is much easier to use!

DragonHunter274 Feb 27

@whitequark I use pixiecore for PXE, its a single command to boot a machine, no DHCP setup needed

@whitequark When I was doing PXE stuff I learned (cant find a reference now) that the PXE Specs contain conflicts. So if one implements/uses PXE one has to choose which part of the spec to break.

I think that says all about that. PXE is a great tool. But it breaks if you look at it wrong.