Mastodawn

Ariadne Conill 🐰

redhat has benefitted from centos and the other clones immensely by the fact that an entire generation of SREs trained on their distro. entire businesses (like my old one) were built around partnering with redhat and providing support and consulting for RHEL and the clones.

the redhat partner network was bootstrapped on the back of the clones. you lab with the clones and then start working with the real thing.

and yes, sometimes users who wanted support went with consultancies like mine and not upgrading to RHEL, it’s true. but that does not matter because when consultancies like mine were working on large contracts with large commitments we would suggest that RHEL be used instead of clones so that there was a point of accountability.

and really, that is and remains the only reason to buy a commercial Linux system: the contracted accountability in the form of the SLA. if a deployment does not require an SLA, then withholding the product just creates a situation where they will use a different one.

that results in a brain drain: the users who would have stayed in your ecosystem (via a clone), will now go learn a different ecosystem. and this causes you to lose your partner network as consultants retire.

Ariadne Conill 🐰

you will be able to watch this shift by observing the evolution of middleware.

for example, lets talk about, say, cPanel. yes, really: cPanel is still around, and people still use it.

or lets talk about SolusVM. yes, really: that exists too, and for better or worse, it is the backbone of the traditional VPS industry.

today, these are built and deployed on RHEL or, more commonly, the clones.

tomorrow? they will be refactored and deployed on alternatives.

if i were to make a bet, i could see OpenSUSE capturing that entire segment of the RHEL/clone userbase within 2 years. later, those products will likely move more heavily in the direction of containerization, but OpenSUSE gives them a landing pad for the short term.

and when that happens? there goes the bulk of the SREs who got their first taste of SRE work by managing a hosting provider. that remains a *huge* segment of the redhat trained SRE footprint.

Ariadne Conill 🐰

mike's post on LinkedIn, where he says that redhat defines "freeloaders" as people who only buy a minimum amount of RHEL licenses while using the clones heavily is spin.

RH have always hated the existence of the clones. they have used various legal chicanery, arguably in violation of the GPL, to attempt to force customers into moving all of their machines to RHEL, and away from clones.

Ariadne Conill 🐰

i think the difference is that, previously, before IBM, leadership understood the need for the clones to exist (despite hating their existence), and that ultimately the clones did drive tangible value (and sales as I outlined above) for RH.

now IBM wishes to manage RH licensing as if it were software for a mainframe. this mentality will kill whatever is left in the product.

Ariadne Conill 🐰

i know so many people who labbed their entire RHCE certification on CentOS that it is not even funny.

when i say the redhat partner network was built on the backs of the clones, i mean this literally.

Anisse Jul 2, 2023

@ariadne
From Red Hat's PoV the training and certification use case is "solved" with the tens of free licenses for individual use.
What they don't see: small businesses training the future generation of SREs by using only clones, will instead be trained on Ubuntu, Debian or whatever.
What's even worse: I think this strategy might succeed in the short and mid term. Numbers will go up. But the reputational damage will be higher than they can measure, and it will be the reason for their stagnation then demise in the long term.

Ariadne Conill 🐰

@Aissen yes, their numbers will go up for a year or two, but by the time RHEL10 happens, it will be tanking.

Zi Real VIPI Jul 2, 2023

iBM has nothing to do with the old rotten hatred of RH exécutives towards clones.

Internet has still the records of interviews of RH top sales getting nuts about centos.

IBM is not an angel but here it bears zero responsibility.

Ariadne Conill 🐰

@zirealvipi sure, but with IBM's takeover, the pressure to maximize revenue is heavier.

Bálint Szilakszi Jul 2, 2023

@ariadne containers will kill all remaining RHEL value

Ariadne Conill 🐰

@szbalint hence why the business has largely shifted toward OpenShift in the past decade

Chupacabra 78704 Jul 2, 2023

@szbalint @ariadne I use Fedora. Always have since RH started charging money. Wasn't that the 90s?

Arcticulate Jul 2, 2023

@ariadne This is the best analysis I’ve heard so far about this subject. Note: I have not read interviews and not done enough research on the topic to just nod without hesitation, I think it makes 100 % sense given everything else I’ve learned about Red Hat’s business model over an extremely long time (Been a user on and off since I first installed Red Hat Desktop Linux 5.2 back in the late ’90s, plus several years RHEL at work).

Ben Ramsey Jul 2, 2023

@ariadne @pchestek I wonder how much competition with Amazon Linux factors into this decision. Maybe it doesn’t, but from personal experience, if you’re running in AWS and you want a hardened, Amazon-supported system, you use Amazon Linux, which is a RHEL derivative.

jollyrogue Jul 2, 2023

@ariadne The problem is Oracle. It’s my understanding RH helped the clones quite a bit, but this all started due to Oracle Linux.

Ariadne Conill 🐰

actually, from what I have heard, the problem is Rocky in this case 🙃

yes, for a while, RH embraced the "community" clones. but that was because there really was not any choice.

but, now, they would rather those users become perpetual beta testers for the next RHEL release instead.

as for Oracle and the other commercial clones: yes, there has been a lot of friction there, but Oracle fought back and as far as I understand, there has just been a quiet truce in recent years

Ariadne Conill 🐰

@jollyrogue i can point to interviews with RH executives back in the 2000s when CentOS 3/4 were still the latest versions, where they were complaining that CentOS was a thing 🙃

jollyrogue Jul 2, 2023

@ariadne True. The clones were good marketing, and I’m not disagreeing with that.

Interesting about Oracle.

Is it Rocky? They seem to be the least problematic. 😆 The new clones are all iffy all around, and I wasn’t going to touch them unless I was forced to, like GitLab did for a couple of my test boxes.

Ariadne Conill 🐰

@jollyrogue yes, the problem is that Rocky is taking away users who they would like to redirect onto Stream.

Clemens Jul 2, 2023

@ariadne @jollyrogue I don't think it's just that; rebuilders are also taking users that would otherwise buy RHEL because they need certifications.

See FIPS for example. Getting certified is expensive. A huge amount of developer time goes into making the crypto libs compliant, the rebuilders just take that effort and re-submit. I have no way to confirm this directly, but I've heard rebuilders going around doing sales marketing with their certification (that they don't yet have).

Clemens Jul 2, 2023

@ariadne @jollyrogue Take a look at the "Rocky Linux 8.6 OpenSSL Cryptographic Module" submitted at https://csrc.nist.gov/Projects/cryptographic-module-validation-program/modules-in-process/iut-list.

Getting crypto FIPS-certified is what currently pays my salary, so I know a thing or two about it — but I have no idea how they plan to get OpenSSL on 8.6 (which is OpenSSL 1.1) certified under FIPS 140-3; that would require a huge amount of dev effort and many patches in my opinion. Where is that development effort happening? It's not here: https://git.rockylinux.org/staging/rpms/openssl/-/commits/r8

Implementation Under Test List - Cryptographic Module Validation Program | CSRC | CSRC

The IUT list is provided as a marketing service for vendors who have a viable contract with an accredited laboratory for the testing of a cryptographic module, and the module and required documentation is resident at the laboratory. The CMVP does not have detailed information about the specific cryptographic module or when the test report will be submitted to the CMVP for validation. When the lab submits the test report to the CMVP, the module will transition from the IUT list to the MIP list. If you would like more information about a specific cryptographic module or its schedule, please contact the vendor.

CSRC | NIST

Clemens Jul 2, 2023

@ariadne @jollyrogue That leaves me wondering: are they just planning to get 140-3 certification for what RHEL ships and certified under 140-2? I don't think that's possible.

I may be wrong, but if I'm not: do they know that this is doomed to fail, and if they do, are they telling their customers that they never really expect to get that certification, or do they just happily go around and tell potential customers and users "yeah, it'll come eventually, just buy support now".

Clemens Jul 2, 2023

@ariadne @jollyrogue https://forums.rockylinux.org/t/fips-validation/4803/26 asks this exact question, but there has never been an answer.

FIPS Validation

Brian, I see that Rocky 8 OpenSSL is still IUT for FIPS 140-3. What version of Rocky 8 openssl is currently being tested for CMVP? RHEL8 is maintaining their 1.1.1 certification under the FIPS 140-2 requirements, but CMVP is longer accept new submissions for 140-2. With the RHEL8 upstream being OpenSSL 1.1.1, I am assuming that Rocky has submitted 1.1.1 for FIPS 140-3, but I have heard that there were new FIPS 140-3 requirements that may not be able to be met with OpenSSL 1.1.1. If Rocky is ...

Rocky Linux Forum

Stéphane Benoit👽Jul 2, 2023

@ariadne RH based distros are a pain compared to the debian based ones, we switched over 15 years ago and never came back

Ariadne Conill 🐰

@stefb sure. and a lot of what i brought to alpine was similarly built on using RHEL in anger.

Stéphane Benoit👽Jul 2, 2023

@ariadne yep, for containers we use mostly alpine too when possible

Ariadne Conill 🐰

@stefb the real value proposition for Alpine is using it on the hosts, tbh

Stéphane Benoit👽Jul 2, 2023

@ariadne didn't know that, i've always used it within containers

qgty Jul 2, 2023

@ariadne I prefer the alternate phrasing of

"Freeloaders are people who don't buy what they don't need"

kouhai, of the health issues Jul 2, 2023

@ariadne good point, perhaps I should give OpenSUSE a try

(that, or alpine, I guess)

the vessel of morganna Jul 2, 2023

@kouhai @ariadne OpenSUSE seems like a pretty good candidate for where SolusVM and Cpanel folks will go. A lot of people operating those systems don't particularly know much about the underlying OS nor care to (hence why they use a turnkey solution in the first place), so they want something that's generally very low maintenance and Just Works:tm:

kouhai, of the health issues Jul 2, 2023

@astraleureka @ariadne ~~I’m also in that category, as I just want a convenient host for my containers~~

evasb 🇺🇦Jul 2, 2023

@ariadne Many people can't make the ends meet if they use RHEL, it is not cheap. They can't ignore these people and say that they are simply "hobbyists", they aren't. The fact that they can't pay for RHEL (atm) doesn't make them hobbyists.

Red Hat sadly refuses to make a solution that would work for both sides, so the clones will continue, and rightfully so.

cro Jul 2, 2023

@ariadne Yes, SUSE is too smart to miss the opportunity presented by RedHat.

🌈 Anna (Spots) cutely Jul 2, 2023

@ariadne something a lot of people seem to be ignoring is that they are also somewhat preventing redistribution of GPL software.

…oh, and by closing off the other branches, it just means downstreams that genuinely needed security/other patches are going to be left insecure because they probably won't even know there are security issues with what they are running. I mean, all the RH security errata are only on the support and partner networks now, they aren't public either…

I'm quite fearful that the fallout of this will be, in the near-term, more unnecessary exploits. long-term, probably the erosion of the RH ecosystem in general, as you noted.

Adrian O’Connor Jul 2, 2023

@ariadne this is a great thread. The company I’m working with right now have just moved away from RHEL, they seem happy to just accept the default options on cloud VMs/containers and don’t seem to mind the lack of SLA on Linux itself.

Matteo Panella Jul 2, 2023

@ariadne There's a big big elephant in the room nobody (especially Red Hat) is talking about: a lot of public research institutions run EL derivatives, licensing costs for RHEL would be higher than their IT budget (sometimes even higher than their *entire* budget) and essentially gobble up public funds that would be better spent for actual research.

Previously I worked in such an institution: we had ~1500 dual-socket physical systems and ~3000 virtual machines in one datacenter alone, all running CentOS or Scientific Linux. One day, Red Hat asked to meet us and tried to sell RHEL with the usual bullshit about CentOS/SL being essentially "stolen work", when we asked if they had discounted licenses for non-profit research institutions they replied with the publicly listed prices for RHEL. Given our numbers, that would have put yearly operating costs in the range of tens of millions of euros just to bless our machines with a license.

We laughed hard and told them they were wasting their time if they thought we'd give them that much money for basically nothing in return ("no, we don't support that" was their recurring answer when we told them what kind of workload was running on those systems).

Toasterson (Till Wegmüller)Jul 2, 2023

@rfc1459 @ariadne Ah, the good old Big corp. We can't find out why our competition is so sucessfull so lets squash it.

Matteo Panella Jul 2, 2023

@ariadne IIRC they now have a heavily discounted offering for some research institutions, but when I asked my former colleagues a few years ago they told me they were still running CentOS on the main batch farm because putting up with Satellite, subscription-manager et. al was a royal pain in the ass.

Also, the discount only covered a few hundred subscriptions out of thousands of systems, so... yeah.

Dźwiedziu Jul 2, 2023

@rfc1459
Let me tell you how the last upgrade of Satellite went at my corpo, and how RH botched the post-mortem support…

…well I can't, because NDA, but you get the picture.

Έλλεν Εμίλια Ά.ζ.Jul 2, 2023

@rfc1459 @ariadne some stuff like the end of Scientific Linux caused probably many orgs to look for alternatives, e.g. TUC switched to Debian (for now at least).

Paul Gessinger Jul 2, 2023

@rfc1459 #cern and the entire LHC computing grid come to mind.

syn Jul 2, 2023

@rfc1459 @ariadne

Troed Sångberg Jul 2, 2023

@rfc1459 Did the systems need to be RHEL-compatible? If so, why?

Ankit Pati Jul 2, 2023

@rfc1459 @ariadne Why can’t they just switch to CentOS Stream?

Halla Rempt Jul 2, 2023

@rfc1459 @ariadne Sounds a lot like the discussion I had with a @qt representative when moonlighting for Quby. No give at all, the deal was, you pay up in full, and then pay some more.

lj·rk Jul 2, 2023

@rfc1459 @ariadne We mostly use(d) Debian, also because CentOS regularly lags behind in security/critical updates, after every RHEL release cadence for up to a few months. What's the reason behind using CentOS in production? I'd argue that even CentOS Stream is more useful here since it's actually up-to-date w.r.t. vuln patches?

Francesco P Lovergine

@rfc1459 @ariadne to be honest in a lot of cases Debian distribution Is more than enough in research environments with any decent IT team. And public institution could return a lot of value to the community. When we started #debiangis blend almost 20 ys ago, Debian was the only ecosystem where all that could happen. Never considered RH or its downstreams.

James Gilbert Jul 2, 2023

@rfc1459 @ariadne We used to run RHEL in our data center. When IBM bought them in 2018 we no longer qualified as an academic institute because, by IBM's rules, we don't directly award degrees. The licensing would have been too expensive, so we have switched to Ubuntu.

Kevin P. Fleming Jul 2, 2023

@rfc1459 @ariadne Did you consider moving to Debian? I'm curious why CentOS/Scientific Linux were the best options for those servers (or Rocky/AlmaLinux for more recent deployments).

If large institutions find some value in using an EL derivative instead of a completely-free LTS distribution, it seems reasonable to me that they shouldn't expect to receive that value at no cost.

DrYak Jul 2, 2023

@kevin switching to either Debian Stable or LTS versions of Ubuntu is literally what is going to happen next at a lot of academic centers.

The only drawback is some specific hardware with blobs that were mostly tested against RHEL only until now (I am thinking e.g stuff related to the cluster fabric, or some storage solutions).

CentOS was a good bet that your Infiniband or Dolphin PCIe interconnect, and proprietary high performance clustered file system will work as intended.

DrYak Jul 3, 2023

@kevin It's not that academic institution get any direct value from EL.
Their usual use case (insane high speed, storage, etc. but availability: mostly recovery within day(s)) is different from the typical EL high paying customer (bank, etc.: mid-range performance, but 6 sigma of reliability and failover in fraction of seconds), and usually EL answers are "we don't cover that".

BUT RHEL has become a de facto standard in business settings, hence hardware support guarantees I mention.

erstwhile Jul 2, 2023

@rfc1459 @ariadne yes..

sad but true, Education is a similar bag, just at smaller scale. My ORG previously ran Scientific Linux too, for many internal services and software not requiring a "Supported" OS by the software vendor

When that folded we went CentOS. When that folded, against advice, the 'org' chose Oracle Linux. If Larry is due payback for harming DB2 sales, its' move again...

I'm guessing paying customers will get love, as they buy binary support, and presumably updates, but expect many systems, it'd be move to paid OEL or pick another horse, I'm guessing a lot of those boxen will become Debian base as the org transitions away.

The org just does not have that budget, in much the same way it sounds your org didn't.

Seriously hoping Rocky finds a stable way forward, but I'd expect the *EL ecosystem may be permanently damaged by this.

I think my Org will now be very reticent of any 'Corporate' Linux vendor with a 'Free' distribution, which is pretty sad.

Matt Jul 2, 2023

@ariadne @rfc1459 let’s not forget that if you pay RedHat and then call RedHat for RHEL support, they can rarely help b/c they mostly only know how to download stuff for free off the internet and then sell it to mega corps

steven Jul 2, 2023

@rfc1459 @ariadne Institutions like this will definitely need to look elsewhere. The EL clone space is so unsettled, and that's exactly what users do not want.

I think a 10-year support life is asking for trouble, and it's possible to get 5 years out of #Ubuntu, #openSUSE and #Debian. Maybe #CentOS Stream, but that's still under RH.

@ariadne is the benefit of RedHat over free linux distros in the support? I have worked in places with thousands of servers that all ran free distros maintained by an in-house team of ops; I never quite understood what benefit a license fee would add ontop of that.

Ariadne Conill 🐰

@carbontwelve the value is the SLA. or, more bluntly, subscribers are paying to have a "neck to choke."

Henri Jul 2, 2023

@ariadne Meanwhile Debian all these years:

MVNDVS PATEAT Jul 2, 2023

@ariadne And all the responsible adults in the room who knew this, were just laid off. Ever so conveniently..

Anyway i wonder what this does to the HPC market no way those folks are going to buy RHEL licenses for 10k+ machines.