jdkoeckThoughts on Slowing the Fuck Down
https://mariozechner.at/posts/2026-03-25-thoughts-on-slowing-the-fuck-down/
> it sure feels like software has become a brittle mess, with 98% uptime becoming the norm instead of the exception, including for big services
As somebody who has been running systems like these for two decades: the software has not changed. What's changed is that before, nobody trusted anything, so a human had to manually do everything. That slowed down the process, which made flaws happen less frequently. But it was all still crap. Just very slow moving crap, with more manual testing and visual validation. Still plenty of failures, but it doesn't feel like it fails a lot of they're spaced far apart on the status page. The "uptime" is time-driven, not bugs-per-lines-of-code driven.
DevOps' purpose is to teach you that you can move quickly without breaking stuff, but it requires a particular way of working, that emphasizes building trust. You can't just ship random stuff 100x faster and assume it will work. This is what the "move fast and break stuff" people learned the hard way years ago.
And breaking stuff isn't inherently bad - if you learn from your mistakes and make the system better afterward. The problem is, that's extra work that people don't want to do. If you don't have an adult in the room forcing people to improve, you get the disasters of the past month. An example: Google SREs give teams error budgets; the SREs are acting as the adult in the room, forcing the team to stop shipping and fix their quality issues.
One way to deal with this in DevOps/Lean/TPS is the Andon cord. Famously a cord introduced at Toyota that allows any assembly worker to stop the production line until a problem is identified and a fix worked on (not just the immediate defect, but the root cause). This is insane to most business people because nobody wants to stop everything to fix one problem, they want to quickly patch it up and keep working, or ignore it and fix it later. But as Ford/GM found out, that just leads to a mountain of backlogged problems that makes everything worse. Toyota discovered that if you take the long, painful time to fix it immediately, that has the opposite effect, creating more and more efficiency, better quality, fewer defects, and faster shipping. The difference is cultural.
This is real DevOps. If you want your AI work to be both high quality and fast, I recommend following its suggestions. Keep in mind, none of this is a technical issue; it's a business process isssue.
It also seems like massive consolidation has caused issues too. Everyone is on Github. Everyone is on AWS. Everyone is behind cloudflare. Whenever an issue happens here it effects everyone and everyone sees it.
In the past with smaller services those services did break all the time, but the outage was limited to a much smaller area. Also systems were typically less integrated with each other so one service being down rarely took out everything.
It's a systems engineering job. You need to provide context, acceptable failure modes, and test at each level for validation. Identify false coupling, poor interfaces, things that don't match business context during agent planning phase. Then communicate / translate to others so their decisions improve instead of destroying the system by optimizing only for their local situation.
I suppose everyone on HN reaches a certain point with these kind of thought pieces and I just reached mine.
What are you building? Does the tool help or hurt?
People answered this wrong in the Ruby era, they answered it wrong in the PHP era, they answered it wrong in the Lotus Notes and Visual BASIC era.
After five or six cycles it does become a bit fatiguing. Use the tool sanely. Work at a pace where your understanding of what you are building does not exceed the reality of the mess you and your team are actually building if budgets allow.
This seldom happens, even in solo hobby projects once you cost everything in.
It's not about agile or waterfall or "functional" or abstracting your dependencies via Podman or Docker or VMware or whatever that nix crap is. Or using an agent to catch the bugs in the agent that's talking to an LLM you have next to no control over that's deleting your production database while you slept, then asking it to make illustrations for the postmortem blog post you ask it to write that you think elevates your status in the community but probably doesn't.
I'm not even sure building software is an engineering discipline at this point. Maybe it never was.
Maybe back in the beginning, but I don't think it's an engineering discipline now. I don't think that's bad though. I always thought we tagged on the word "engineer" so that we could make more money. I'm ok with not being one. The engineers I've known are very strict in their approach which is good since I don't want my deck to fall down. Most of us are too risky with our approach. We love to try new things and patterns, not just used established ones over time. This is fine with me, and when we apply the term "engineer" to work, I get a little uneasy, because I think it implies us doing something that most of us really don't want to do. That is, absolutely prove our approach works and will work for years to come. Just my opinion though.
I’ve had jobs where my title was “software engineer”, but I never refer to myself as such outside of work. When I tell others what I do, I say I am a software developer. It may seem a pointless distinction, but to me there is a distinction.
Neither myself nor the vast majority of other “software engineers” in our field are living up to what it should mean to be an “engineer”.
The people that make bridges and buildings, those are the engineers. Software engineers, for the very very most part, are not.
I was won over by this distinction from another senior some years ago. I think he said…
“Developers build things. Engineers build them and keep them running.”
I like the linguistic point from a standpoint of emphasizing a long term responsibility.
I was just reading "how the world became rich" and they made an interesting distinction economic "development" vs plain "growth". Amusingly, "development" to them means exactly what you're saying "engineer" should mean. It's sustainable, structural, not ephemeral. Development in the abstract hints at foundational work. Building something up to last. It seems like this meaning degradation is common in software. It still blows my mind how the "full-stack" naming stuck, for example.
https://www.howtheworldbecamerich.com/
Edit-on a related note, are there any studies on the all-in long-term cost between companies that "develop" vs. "engineer". I doubt there would be clean data since the managers that ignored all of the warning of "tech debt" would probably have the say on both compiling and releasing such data.
Does the cost of "tech-debt" decrease as the cost of "coding" decreased or is there a phase transition on the quality of the code? I bet there will be an inflection point if you plotted the adoption time of AI coding by companies. Late adapters that timed it after the models and harnesses and practices were good enough (probably still some time in the near future) would have less all-in cost per same codebase quality.
I like this one from Russ Cox:
"Software engineering is what happens to programming when you add time and other programmers."
I'm similar except for me reason is no degree. So some jobs eng others just developer... although my current job I'm a "technology specialist" which is funny. But I'm getting paid so whatever.
Most recently I wrote cloudformation templates to bring up infra for AWS-based agents. I don't use ai-assisted coding except googling which I acknowledge is an ai summary.
A friend of mine is in a toxic company where everyone has to use AI and they're looked down upon if they don't use it. Every minute of their day has to be logged doing something. They're also going to lay off a bunch of people soon since "AI has replaced them" this is in the context of an agency.
Are We Really Engineers?
This is part one of the Crossover Project. Part two is here and part three is here. A conference talk based on this work is now available here. I sat in front of Mat, idly chatting about tech and cuisine. Before now, I had known him mostly for his cooking pictures on Twitter, the kind that made me envious of suburbanites and their 75,000 BTU woks. But now he was the test subject for my new project, to see if it was going to be fruitful or a waste of time.
It’s a bit of a misclassification. In my mind we tend to be more like architects where there are a fair amount of innovative ideas that don’t work all that well in practice. Train stations with beautiful roofs that leak and slippery marble floors, airports with smoke ventilation systems in the floor, etc.
Of course, we use that term for something else in the software world, but architecture really has two tiers, the starchitects building super fancy stuff (equivalent to what we’d call software architects) and the much more normal ones working on sundry things like townhomes and strip malls.
That being said I don’t think people want the architecture pay grades in the software fields.
At the same time, if you remove 'engineer' , informatics should fall under the faculty of Science, so scientists, which are even more rigorous than engineers ;)
Maybe software tinkerer?
It's a Systems Engineering job. You provide context, define interfaces to people, tests for critical failure modes affecting customer, describe system behavior, and translate to other people.
People don't realize how much software engineering has improved. I remember when most teams didn't use version control, and if we did have it, it was crappy. Go through the Joel Test [1] and think about what it was like at companies where the answers to most of those questions was "no."
[1] https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
At the same time, systems have become far more complex. Back when version control was crap, there weren't a thousand APIs to integrate and a million software package dependencies to manage.
Sure everything seems to have gotten better and that's why we now need AIs to understand our code bases - that we created with our great version control tooling.
Fundamentally we're still monkeys at keyboards just that now there are infinitely many digital monkeys.
Perrow’s book Normal Accidents postulates that, given advances which could improve safety, people just decide to emphasize throughout, speed, profits, etc. he turned out to be wrong about aviation (got much safer over time) and maritime shipping (there was a perception of a safety crisis in the late 1970s with oil tankers exploding, now you just hear about the odd exceptional event.)
> Perrow argues that multiple and unexpected failures are built into society's complex and tightly coupled systems, and that accidents are unavoidable and cannot be designed around.[1]
This is definitely something that is happening with software systems. The question is: is having an AI that is fundamentally undecipherable in its intention to extend these systems a good approach? Or is an approach of slowing down and fundamentally trying understand the systems we have created a better approach?
Has software become safer? Well planes don't fall from the sky but the number of zero day exploits built into our devices has vastly improved. Is this an issue? Does it matter that software is shipped broken? Only to be fixed with the next update.
I think its hard to have the same measure of safety for software. A bridge is safe because it doesn't fall down. Is email safe when there is spam and phishing attacks? Fundamentally Email is a safe technology only that it allows attacks via phishing. Is that an Email safety problem? Probably not just as as someone having a car accident on a bridge is generally not a result of the bridge.
I think that we don't learn from our mistakes. As developers we tend to coat over the accidents of our software. When was the last time a developer was sued for shipping broken software? When was the last time an engineer was sued for building a broken bridge? Notice that there is an incentive as engineer to build better and safer bridges, for developers those incentives don't exist.
Version control is useful but it has nothing to do with software engineering per se. Most software development is craft work which doesn't meet the definition of engineering (and that's usually fine). Conversely, it's possible to do real software engineering without having a modern version control system.
> I'm not even sure building software is an engineering discipline at this point. Maybe it never was.
If I engineer a bridge I know the load the bridge is designed to carry. Then I add a factor of safety. When I build a website can anyone on the product side actually predict traffic?
When building a bridge I can consult a book of materials and understand how much a material deforms under load, what is breaking point is, it’s expected lifespan, etc. Does this exist for servers, web frameworks, network load balancers, etc.?
I actually believe that software “could” be an engineering discipline but we have a long way to go
Software and bridges are entirely different.
If I need a bridge, and there's a perfectly beautiful bridge one town over that spans the same distance - that's useless to me. Because I need my own bridge. Bridges are partly a design problem but mainly a build problem.
In software, if I find a library that does exactly what I need, then my task is done. I just use that library. Software is purely a design problem.
With agentic coding, we're about to enter a new phase of plenty. If everyone is now a 10x developer then there's going to be more software written in the next few years than in the last few decades.
That massive flurry of creativity will move the industry even further from the calm, rational, constrained world of engineering disciplines.
Software packages are more complicated than you make them out to be. Off the top of my head:
- license restrictions, relicensing
- patches, especially to fix CVEs, that break assumptions you made in your consumption of the package
- supply chain attacks
- sunsetting
There’s no real “set it and forget it” with software reuse. For that matter, there’s no “set it and forget it” in civil engineering either, it also requires monitoring and maintenance.
>I actually believe that software “could” be an engineering discipline but we have a long way to go
It certain mission critical applications, it is treated as engineering. One example - https://en.wikipedia.org/wiki/DO-178B
I think it is in certain very limited circumstances. The Space Shuttle's software seems like it was actually engineered. More generally, there are systems where all the inputs and outputs are well understood along with the entire state space of the software. Redundancy can be achieved by running different software on different computers such that any one is capable of keeping essential functions running on its own. Often there are rigorous requirements around test coverage and formal verification.
This is tremendously expensive (writing two or more independent copies of the core functionality!) and rapidly becomes intractable if the interaction with the world is not pretty strictly limited. It's rarely worth it, so the vast majority of software isn't what I'd call engineered.
> can anyone on the product side actually predict traffic
Hypothetically, could you not? If you engineer a bridge you have no idea what kind of traffic it'll see. But you know the maximum allowable weight for a truck of X length is Y tons and factoring in your span you have a good idea of what the max load will be. And if the numbers don't line up, you add in load limits or whatever else to make them match. Your bridge might end up processing 1 truck per hour but that's ultimately irrelevant compared to max throughput/load.
Likewise, systems in regulated industries have strict controls for how many concurrent connections they're allowed to handle[1], enforced with edge network systems, and are expected to do load testing up to these numbers to ensure the service can handle the traffic. There are entire products built around this concept[2]. You could absolutely do this, you just choose not to.
[1] See NIST 800-53 control SC-7 (3)
[2] https://learn.microsoft.com/en-us/azure/app-testing/load-tes...
The way the authors of the book on material strengths got those numbers, was through testing. If you're using mature technologies, that testing has been done by others and you can rely on it for your design, at least in a general way. Otherwise you have to do the testing yourself, which is something a structural engineering project might do also, if it's unusual in some way.
> What are you building?
This x1000. The last 10 years in the software industry in particular seems full of meta-work. Building new frameworks, new tools, new virtualization layers, new distributed systems, new dev tooling, new org charts. All to build... what exactly? Are these tools necessary to build what we actually need? Or are they necessary to prop up an unsustainable industry by inventing new jobs?
Hard to shake the feeling that this looks like one big pyramid scheme. I strongly suspect that vast majority of the "innovation" in recent years has gone straight to supporting the funding model and institution of the software profession, rather than actual software engineering.
> I'm not even sure building software is an engineering discipline at this point. Maybe it never was.
It was, and is. But not universally.
If you formulate questions scientifically and use the answers to make decisions, that's engineering. I've seen it happen. It can happen with LLMs, under the proper guidance.
If you formulate questions based on vibes, ignore the answers, and do what the CEO says anyway, that's not engineering. Sadly, I've seen this happen far too often. And with this mindset comes the Claudiot mindset - information is ultimately useless so fake autogenerated content is just as valuable as real work.
> The last 10 years in the software industry in particular seems full of meta-work. Building new frameworks, new tools, new virtualization layers, new distributed systems, new dev tooling, new org charts. All to build... what exactly?
Don't forget App Stores. Everyone's still trying to build app stores, even if they have nothing to sell in them.
It's almost as if every major company's actual product is their stock price. Every other thing they do is a side quest or some strategic thing they think might convince analysts to make their stock price to move.
Hey Visual Basic is still there, and last time I checked it was still the goto option to do OLE Automation.
RoR is no longer at its peak, but is still have its marginal stable share of the web, while PHP gets the lion part[1]
Ok, Lotus Notes is really relic from an other era now. But it’s not a PL, so not the same kind of beast.
Well, also LLMs are different beast compared to PL. They actually really are the things that evocate the most the expression "taming the beast" when you need to deal with them. So it indeed as far away as possible of engineering as one can probably use a computer to build any automation. Maybe to stay in scientific realms ethology would be a better starting point than a background in informatics/CS to handle these stuffs.
Useful context here is that the author wrote Pi, which is the coding agent framework used by OpenClaw and is one of the most popular open source coding agent frameworks generally.
That's hilarious. I've been following Mario since his work on libGDX and RoboVM.
His blog post on pi is here: https://mariozechner.at/posts/2025-11-30-pi-coding-agent/
That's a great shout because I'm sure a lot of people would otherwise just discredit this take as just another anti-ai skeptic. But he probably has more experience working with LLM's and agents than most of us on this site, so his opinion holds more weight than most.
If you were going to dismiss an argument because of who it comes from rather than its content, that is a flaw in your thinking. The argument is correct, or it isn't, no matter who said it.
Your ability to evaluate whether the argument is correct is limited. In theory, the author and the correctness of the argument are unrelated, in practice, the degree of experience the author has with the topic they’re making an argument on does indeed have some correlation with the argument and should influence the attention you give to arguments, especially counterintuitive ones.
> Companies claiming 100% of their product's code is now written by AI consistently put out the worst garbage you can imagine. Not pointing fingers, but memory leaks in the gigabytes, UI glitches, broken-ass features, crashes
One thing about the old days of DOS and original MacOS: you couldn't get away with nearly as much of this. The whole computer would crash hard and need to be rebooted, all unsaved work lost. You also could not easily push out an update or patch --- stuff had to work out of the box.
Modern OSes with virtual memory and multitasking and user isolation are a lot more tolerant of shit code, so we are getting more of it.
Not that I want to go back to DOS but Wordperfect 5.1 was pretty damn rock solid as I recall.
Another factor at work is the use of rolling updates to fix things that should better have been caught with rigorous testing before release. Before the days of 'always on' internet it was far too costly to fix something shipped on physical media. Not that everything was always perfect, but on the whole it was pretty well stress-tested before shipping.
The sad truth is that now, because of the ease of pushing your fix to everything while requiring little more from the user than that their machine be more or less permanently connected to a network, even an OS is dealt with as casually as an application or game.
> Modern OSes with virtual memory and multitasking and user isolation are a lot more tolerant of shit code, so we are getting more of it.
It's not the glut of compute resources, we've already accepted bloat in modern software. The new crutch is treating every device as "always online" paired with mantra of "ship now! push fixes later." Its easier to setup a big complex CI pipeline you push fixes into and it OTA patches the users system. This way you can justify pushing broken unfinished products to beat your competitors doing the same.
What the article doesn't touch on is the vendor lock-in that is currently underway. Many corps are now moving to an AI-based development process that is reliant on the big AI providers.
Once the codebase has become fully agentic, i.e., only agents fundamentally understand it and can modify it, the prices will start rising. After all, these loss making AI companies will eventually need to recoup on their investments.
Sure it will be - perhaps - possible to interchange the underlying AI for the development of the codebase but will they be significantly cheaper? Of course, the invisible hand of the market will solve that problem. Something that OPEC has successfully done for the oil market.
Another issue here is once the codebase is agentic and the price for developers falls sufficiently that it will significant cheaper to hire humans again, will these be able to understand the agentic codebase? Is this a one-way transition?
I'm sure the pro-AIs will explain that technology will only get cheaper and better and that fundamentally it ain't an issue. Just like oil prices and the global economy, fundamentally everything is getting better.