Mastodawn

Robert Károly Aug 5, 2024

Ryan Townsend (old account)

👏 more 👏 developers 👏 need 👏 to 👏 hear 👏 this

I can count on one hand the number of my clients over the past couple of years who haven't either over-architected for scale or were unnecessarily concerned about it (prior to coming to me for strategic advice, of course 😉).

You don't need to understand Distributional Little's Law to figure this out, it's obvious with primary school level math.

(screenshot excerpt from https://tailscale.com/blog/new-internet)

The New Internet: Tailscale's Vision for the Future of Connectivity

Tailscale's CEO, Avery Pennarun, explores the future of the internet, discussing the evolution of networking, the issues with today's complex systems, and how Tailscale is simplifying connectivity for developers with a new, more efficient approach. Discover the vision for a New Internet.

@ryantownsend just last week I talked to a client if we would use kubernetes for their new shop.

And I had to explain that "no. that's complete overkill. A single medium sized server is more than enough for what you're going to do"

xchange Jul 27, 2024

@skaverat @ryantownsend the problem is cargo cult everywhere amirite?

cuan_knaggs Jul 27, 2024

@ryantownsend but if you don't keep layering complexity over everything you might have to start addressing the tech debt

mytwobits01 Aug 14, 2024

@mensrea @ryantownsend
But companies are supposed to grow forever! Isn't technical debt part of that??

tsia 🔜 GPN24 Jul 27, 2024

@ryantownsend I moved our matomo instance from VM to Kubernetes with absolutely no autoscaling or performance considerations. Ran a benchmark and hit almost 1000req/s and was like „ok I don’t need to worry about performance of that thing until I die“ 😂

Clemens Jul 27, 2024

@ryantownsend clients probably pay more for products that feel more complex? these fancy sounding software stacks look very reasonable from the business side of things.

Ryan Townsend (old account)Jul 27, 2024

@ceulig I find it's more often driven by developers themselves.

For the past decade+ we've had big tech firms fighting over talent, so they present solutions to their specific problems as panacea to keep the spotlight on them and drive open source adoption.

On the flip side, developers want to believe they will need that level of scale, that they'll witness the same problems and they want their CVs to include the tech that shows they are in the big leagues.

A vicious cycle.

Konnor Rogers Jul 27, 2024

@ryantownsend @ceulig Resume driven development

Salacious Crumb Jul 28, 2024

@konnorrogers @ryantownsend @ceulig It works for lawyers, right?

The world would be a better place if we simplified laws and made them plain language. Except then we wouldn't need lawyers anymore. So they keep it esoteric and use Latin that only means something if you went to law school, therefore perpetuating the need.

wrosecrans Jul 27, 2024

@ryantownsend @ceulig I was 100% on a team at a previous employer where the project was stupid, everybody understood it had no articulated success condition, and everybody but me just sort of stayed quiet and added Kubernetes to their resume. I went slightly mad about the fact that it was pointless and wasteness and nobody else cared. I just got fired from the team for asking useful questions.

Ryan Townsend (old account)Jul 27, 2024

@wrosecrans @ceulig painful 😞 onwards and upwards though!

Accept purpose.Jul 28, 2024

@ryantownsend @ceulig so Kubernetes functions as a signal of inclusive fitness for developers? Attracting potential employment like a peacock's tail attracts peahens.

NSKE Jul 28, 2024

@ryantownsend @ceulig that's it - CV driven development.

Mirco Blitz Jul 27, 2024

@ryantownsend npn install stupidity

Radio Azureus Jul 28, 2024

@lindworm @ryantownsend

<< stupidity.library not found or out of productive development

<< try stdio.h in C

#POSIX #OpenSource #programming #Linux

Laura Orchid Jul 27, 2024

@ryantownsend wow, they did define the meaning of malicious bots/scrapers 😮

Duco Jul 27, 2024

@ryantownsend they all want to be the next Google and Facebook, so they have to scale, when they are successful. 😜

I don't get Kubernetis as it's as complicated as installing software in VMs. There are small benefits and disadvantages.

Somethings that scales and helps developing is serverless though. There you configure when the code is triggered and don't have to care for load balancing, OS updates, monitoring, etc. So there you actually have less work.

Allan Chow Jul 27, 2024

@ryantownsend a lot of engineers suffer from focusing on executing a solution, rather than solving a problem. With that said, going to click the link and read!

Allan Chow Jul 27, 2024

@ryantownsend mm. I do agree with where they're getting at and it will be really cool to see if they unlock startups to self host again. The great MailChimp server migration story being something similar.

I don't think that's the cause for the insane complexity of "just launch a service" comes from though.

Ryan Townsend (old account)Jul 27, 2024

@grumpasaurus fwiw, I see Kubernetes as basically self-hosting, yes it might be atop IaaS but your own team is responsible for uptime, security patching etc.

Most businesses should be outsourcing to a PaaS like Heroku, Render etc.

For 99%+ of use-cases, until your bill hits six figures monthly, it just doesn't make sense to hire DevOps/Infra people.

Allan Chow Jul 27, 2024

@ryantownsend I think it just depends on your situation, use cases, and constraints that a vendor may have on your needs. Of course the most paramount situation being the type of people you have on hand and the budget you have to hire what function you're lacking.

Ryan Townsend (old account)Jul 27, 2024

@grumpasaurus absolutely, context is always important, hence my use of "most businesses" and "99%+"

Ultimately: most applications just aren't special. Many are barely more than simple CRUD.

It's not uncommon that the people on hand are the problem... they are the ones driving the infrastructure complexity in order to keep their jobs and pad their CVs. It can mean swapping them out for more productive people who focus on shipping.

Allan Chow Jul 27, 2024

@ryantownsend I'd like to think that people aren't necessarily being malicious to keep their jobs, but moreso maybe a bit too immature in understanding the complexity of taking on a new-to-them solution, especially when they haven't had the past experience of operating a similar process.

"Let's adopt this CI/CD pipeline so that my job in troubleshooting it is secure" - said no one

Ryan Townsend (old account)Jul 27, 2024

@grumpasaurus not directly, but a DevOps team isn’t going to advocate moving away from Kubernetes/IaaS/bare metal and just using Heroku or Render.

React has seen enormous, continuous adoption despite very apparently flaws because “that’s where the employment/money is” (I’ve researched this before for my conference talks, it was the #1 voted answer on Reddit for why people use it)

Allan Chow Jul 27, 2024

@ryantownsend I think maintaining status quo vs making architectural decisions for a startup are two very different things.

But yeah with react, it's hit that critical mass flywheel of adoption because of its popularity. There are a lot of things that come with that popularity despite its complexity and overhead. I guess reddit is as good a spot as any to gauge industry sentiment.

Allan Chow Jul 27, 2024

@ryantownsend but to go with decisions a startup tends to make, if your founders has a devops person, chances are they're going to go with a particular solution. If your first software developer is a spring person, chances are they're going to go spring. The situational decision making is so dependent on whoever happened to be there to make the first decision and the company is locked for quite a while.

Ryan Townsend (old account)Jul 27, 2024

@grumpasaurus of course they will... I’m here just trying to educate people into thinking a little more critically and commercially.

The reason why I’m often hired is for startups to avoid making these hiring/architectural mistakes in the first place ☺️ …or for more established firms to undo these mistakes so they can actually ship.

Wolf480pl Nov 24, 2024

@ryantownsend @grumpasaurus
not sure if we're using the same definition of "startup", but from what I've read, the investors in those startups are hoping that the company will be the first to a newly discovered market niche, use the first-mover advantage, and take monopoly / dominant position before anyone else notices what's happening. Or that the startup fails fast if it can't do that.

In that case, it seems that any successful startup will need to scale sooner or later.
Though maybe later.

Wolf480pl Nov 24, 2024

@ryantownsend @grumpasaurus
(and also, what investors want might not be what's best for the company)

Wolf480pl Nov 24, 2024

@ryantownsend @grumpasaurus
> a DevOps team isn’t going to advocate moving away from Kubernetes/IaaS/bare metal

Unless the DevOps team is overworked / burnt out, they can barely keep the existing stuff running, and the devs are making a new app that needs unusual stuff, and most new apps in the company die after a quarter - I think in that situation, many DevOps would advocate for running that one app on Heroku if that's what the devs already know.

though that's an extreme case I guess

Ryan Townsend (old account)Nov 24, 2024

@wolf480pl @grumpasaurus fair comment, but I meant completely moving away (given they’d be out of a job)

Wolf480pl Nov 24, 2024

@ryantownsend @grumpasaurus
they'd have a shitton of work for the next year to move the stuff, and then there's still a Jenkins / some other CI that needs to keep running that can sink an infinite amount of effort :P

But yeah fair point, they'd probably prefer to keep status quo.

Wolf480pl Nov 24, 2024

@ryantownsend @grumpasaurus
Also, I think a large problem is that while we teach people to Kubernetes and stuff, we don't teach them why, and when.

It took me a few years of hating k8s from sidelines, and then a couple years of working with k8s at $dayjob, to finally figure out what it's for:

It's for when you have multiple programs and multiple computers and you have trouble figuring out which program should run on which computer.

How many companies would avoid k8s if they knew that?

Allan Chow Nov 24, 2024

@wolf480pl @ryantownsend yeah you really need someone with either a lot of foundational background to learn k8s (like early in its inception) or someone who has a lot of operational background in k8s to know its kinks to help the team get onboard

Wolf480pl Nov 24, 2024

@grumpasaurus @ryantownsend
I mean, if you tell people what kubernetes is for, and they realize they have one computer... or many computers but they want to run the exact same programs on each of them, then they might realize they don't need kubernetes.

Allan Chow Nov 24, 2024

@wolf480pl @ryantownsend yeah that happens quite often! Not too long ago a company I worked at was running bare metal and keeping the business going and in a huge multi year scaling initiative, some architects chose to adopt an entirely different stack expecting to run in AWS despite having no experience running in AWS and the stack itself.

In those situations, where there's timeline and technological cliffs that the company is barreling towards, decisions to buy into something you may not understand but gives you the confidence that it *just works* are extremely easy to make.

Jay Little Jul 27, 2024

@ryantownsend God I wish I knew. Modern software development is such a drag and productivity levels are so low. Every year it seems to get worse and worse too.

I used to literally code features while in the meetings the requests were being relayed to me. This was with ASP Classic web apps in the 2000 - 2004 time span.

Now we hear about some half-baked thing that won't be fully fleshed out until after we push code to clients and it takes months upon months to build.

#OldSchool #GetOffMyLawn

Cybarbie Jul 27, 2024

@ryantownsend It's the old premature optimisation over and over again with some people. If you look at Discord currently shifting trillions of messages; they didn't scale, they started with Mongo, good choice back then and easy to get to market on but could scale up only so far, so they went cassandra but when some of their biggest nodes with enormous channels began to choke they went to scylla. They managed the change!

lick here for more info Jul 27, 2024

@ryantownsend i handle an ecommerce website that serves 500k requests a day, with about a half of them dynamic (i.e. at least a few DB requests). On a single cheap dedicated box.

In fact I'm quite certain almost all of them take 50ms or less because it's one single box. Because I can run a hundred simple database queries instead of one ballooning one. And then put that in a shared, centralized cache. And just invalidate the whole damn cache when any actual meaningful writes happen. Which are minutes apart even during business hours.

Yes, if that server goes down in flames, I'll have to drop everything and migrate, and there will be several hours of downtime. That's gonna suck, I guess, but until then the only downtime we ever had was when (a different, thankfully) data center literally went up in flames and there were some temporary network issues.

Oh, and I guess when debian-security decides to update at 6 am I sometimes get an error report or two while Postgres is restarting.

Turns out if you don't make your shit out of a thousand brittle pieces in the name of "resilience", it just doesn't tend to fall apart on its own. Who knew.

Ryan Townsend (old account)Jul 27, 2024

@virtulis my last startup was in eCommerce too, we saw spikes of thousands of requests/sec and served it all from Heroku without any issue.

I had clients doing 9-figure revenues and I had no need for any DevOps staff.

Mikalai Jul 28, 2024

@virtulis @ryantownsend
I personally started k8s when minikube have appeared so that on a local machine we can simulate the whole op thing, or as much as possible. You still need a db for non-static site, and it is a different process, and you want to test it, locally and fast with same internal network setup as in production.

But then, I colocate pods on the same host for speed. And lower bill.

K. N. Brindle (they/them)Jul 27, 2024

Yup. And while they’re spending all that time on service scaling, they’re often ignoring the much more important part of architecture: writing code systems that are easily readable and understandable at both the function level and the system level. When I was a software architect, that was my biggest battle: keeping senior devs from writing inscrutably complex systems that the junior devs couldn’t grok, and which immediately descended into a tangle of spaghetti

Jfrench Jul 27, 2024

@ryantownsend the only people who want to run at scale are those who aren't. It's much less fun having real scalability problems. Which are usually disks filling up and database contention.

✨ Mia’s Simulacrum 🏳️‍⚧️✨Jul 27, 2024

This is funny - but kind of comparing apples with oranges.

There are just websites that are incredibly resource hungry.

Like google, YouTube, maps or anything that delivers great amounts of dynamic data to users.

And to brag, I could serve this manny users with a static page per hour on a 0.5W ASIC 💅💅💅
A phone is total overkill.

Ryan Townsend (old account)Jul 27, 2024

@m of course: context always matters.

But the point stands: *so many* developers hugely over-complicate things, not even just infrastructure (see also: React and SPA frameworks).

✨ Mia’s Simulacrum 🏳️‍⚧️✨Jul 27, 2024

For personal and small companies absolutely true.

For large companies i would argue that this is wanted since you can spread out complexity this way.

For example I could build all relevant features of kubernetes for auto health and rolling deploy in a single binary of a little self written web server and could even add a dns load balancer to it for good measure.
But this means that a company has many not standardized code and methods for the same thing and needs good developers and good documentation.
It is much better to spread this complexity onto highly specialized ppl that for example only do AWS and Kube, but they can do it for everything.

And since it just takes time to add a feature like auto scaling to such a project it is just cheaper to do it from the beginning because the additional complexity is not that big when you have already everything in place to do it.

It is, I think the was capitalism forces companies to rationalize complexity and IT ppl into conveyor belt work.

To illustrate: you can build a 0.25 cars in your garage per year, but a company that builds already a lot of cars in their factory hall will not hire you or build a garage to produce a new variation, they tell 200 ppl to change a little move different.

Aka over engineering costs less money if you are big enough :P

(Also I think the Tailscale thingy was just a project one would put on a CV to show that one can do the kubernetes specialization)

Andres Jalinton Jul 27, 2024

@m @ryantownsend
Exactly that, how are they comparing kuberenetes to... python and perl?
I get the idea that software can be bloated and there is no need for massive infrastructure for it to serve lots of requests (let me doubt about that absolute remark) but scaling tools have nothing to do with the underlying software efficiency.

Amᵃᵖanda | map data witch Jul 28, 2024

@m @ryantownsend you would be surprised how you can run many OpenStreetMap maps on just a regular single beefy server

lolcat Jul 27, 2024

Hadoop, man.

Somehow, millions of people were convinced to waste a decade and billions of dollars by trying to do everything with Hadoop. FWIW, the overlap between the Hadoop zealots and the LLM zealots is almost perfect.

nivs Jul 27, 2024

@ryantownsend @baldur Me, running my blog on Kubernetes.

Ryan Townsend (old account)Jul 27, 2024

@nivrig @baldur classic 👌🏻

nivs Jul 27, 2024

@ryantownsend @baldur Our production system is a couple of AWS instances and MariaDB. Recently we resized the DB to a larger instance. It hasn’t really needed to scale in 8? years apart from that, and has been acquired twice in that time.

nivs Jul 27, 2024

@ryantownsend @baldur OTOH my previous job did use Kubernetes, because it made managing frequent deployment of their 80+ applications easier, and some of them actually had significant traffic spikes at times that needed autoscaling to handle. Tools for jobs.

FErki Jul 27, 2024

@ryantownsend I designed/built/ran infra where we considered 100k/s requests a nice quiet period, with ~350k/s daily peaks, plus ingest ~1TB/hour data.

Up to this scale, we never found a challenge for which cloud, containers, or Kubernetes would have been the simplest solution that could work – also far from cheapest.

All bare metal on #Gentoo. Managed with #Rex. Globally from ~30 to ~400 servers, then beyond.

Solve problems _you_ have. Not what others think you should solve.

Ryan Townsend (old account)Jul 27, 2024

@ferki wow those are some serious numbers!

I should get your last sentence as a tattoo at this point (to remind others… not myself 😅)

FErki Jul 27, 2024

@ryantownsend Hmm, perhaps I should pivot into doing a swag/merch/lifestyle business instead of performance consulting! 🤔 🤣

Szymon Nowicki Jul 27, 2024

@ferki @ryantownsend IMHO scaling is just one of the k8s features. The biggest problem it solves is to orchestrate micro services into something one can have control over.

Then, micro services main problem to solve is multiple teams cooperation in a way they don’t want to kill each other.

It was never a “technical” choice, although many sold it that way.

FErki Jul 27, 2024

@hey @ryantownsend Exactly! When Kubernetes is the simplest solution to one's own challenge, by all means, please do invest into using Kubernetes 👍

If it's not the best, then invest into understanding.

My point is to encourage understanding the problem before choosing a fitting solution, instead of trying to fit the problem to a chosen solution :)

Most teams I support need help due to a mix of such misalignment, lack of understanding their stack, and failing to shift their infra when needed.

Martijn Vos Jul 27, 2024

There are websites that are enormously resource hungry, and I'm working on one, but the vast majority of websites are not. But they are often very labour intensive because of this.

Honestly, just look at Twitter; they started out in Ruby without any concern for performance or scalability, and only when it got popular and the site kept going down, they rewrote the whole thing in Scala with performance in mind. But at that time, they'd figured out what the site was supposed to do.

Remember what Knuth said about premature optimization.