I think we're focused on the wrong thing when we look at what tech works for a company like Amazon or Facebook or Netflix.

We should be looking at what tech works when you *don't* have a small army of staff engineers optimizing it. I want to know what I can scale *without* paying someone a half million dollar salary to do it.

There should be more case studies on things that don't have a billion-dollar company propping them up, humming along quietly on a cheap-ass VPS somewhere.

@collinsworth omg 100%. I used to have this conversation a lot with folks. It's true of a lot of things: the tech they use, the processes they follow, the way that they allocate time, etc.

Stay nimble, watch costs, and focus on what will make you and your team successful. 🔥

@jeffpalmer @collinsworth Getting a bit cynical, a lot of business tech case studies are written by the PR department of some vendor with a product to sell.

Of course they're going to focus on large enterprise-scale customers with big budgets to do a custom deployment of their product.

And of course they're not going to highlight how a small- or medium-sized business can get away with an some self-hosted network drives, or some WordPress plugin that basically achieves the same thing...

@collinsworth @danilo this is the kind of tech that I try to build in my open source work, and… it’s really hard, and people *say* they want it, but it feels like swimming upstream. The nine-layer architecture diagrams suck up all the oxygen in the room
@collinsworth @danilo “why call a function in a library when instead you could spin up a kubernetes pod that consumes $1500/mo of AWS spend”

@glyph @collinsworth @danilo > “why call a function in a library when instead you could spin up a kubernetes pod that consumes $1500/mo of AWS spend”

This is a question I've asked so many times at so many companies, though phrased slightly differently (why does this have to be a micro service partitioned by the network and living in a separate repo, instead of a simple function in a module) and hardly ever gotten an intelligent answer to. It generally boils down to perceived needs that never materialize. Or because they think they're following best practices.

All they are doing is increasing accidental complexity. Exactly the thing they should be minimizing. It's frustrating.

@collinsworth nah, Google once wrote some overly optimistic quarterly goals on a whiteboard and then they took over the world, so now we all have to make OKRs, that's just how it works

@collinsworth Having worked in a big (multi-billion dollar, hundreds of millions of concurrent users) I concur so hard.

Scale is a trap.

I am, at this point, something of an expert of doing a lot with very very little, and this is the skillset I am the proudest of.

@ajroach42 @collinsworth we call this "scale" yet this doesn't scale, Big Tech serves primarily shareholders at the expense of everyone else. When we have technology that actually scales it means putting power back in the hands of you and me, serving stakeholders interests, on our terms.
@collinsworth This is one of appeals of writing servers in Rust to me. It has a better chance of just working really well by default. Granted I'm not sure if I could deploy to a VPS though. I think Svelte on the front end is a similar appeal.
@collinsworth 100%. More discussion and case studies would help a lot with evaluating tools against the kind of situations I might actually encounter. On the business as well as the technical side: one of the reasons I’ve been using Laravel is the business model seems much more closely aligned with my goals, giving me hope the direction will be more aligned longer term.
@collinsworth I get what you’re saying, but if people weren’t constantly trying to break in to systems, the need for these knowledgeable people whose salaries are based on the costs of getting the knowledge to some degree. Scaling is easy. Securing the contents is hard. And system updates need to be tested for breaking changes (Windows 11 has some big ones coming up that break a lot of third party tools for instance)
It’s a great idea damaged by bad actors.
@collinsworth totally, the vast majority of tech inside Amazon makes zero sense for anything smaller.

@collinsworth There are. Ask the erlang/elixir folks. We have tales.

I can tell you the 99% result if you want. It gets killed. Because it does not fit the growth view of the world. And not spending money faster is a death sentence for a VC startup.

Also means managers cannot easily use metrics proxy, because you cannot ignore the complexity.

The biggest recent case is Bleacher Report

@Di4na @collinsworth Ericsson is definitely a billion-dollar company. Of course you could debate whether they “propped up” erlang or treated it like shit.
app

A web development kit that’s small, purrs, and loves you.

Codeberg.org
Couldn't agree more. I outsource my email and cloud storage to Proton (who have zero access to the encrypted data), but everything else, including this fediverse instance and my website, run off my own home servers. I spend a few dollars a month to keep them going and it's been working very well so far.

@collinsworth

Me.

I'm this case study.

I'm the guy who built a global >100gbps network spanning 8 locations on 3 continents, based mostly on SME FTTP, secondhand eBay hardware, and a couple of strategic colocated racks.

If you want this case study, talk to me.

[edit: now that VyOS is more mature, I probably would need half the hardware]

@collinsworth @bjoreman I’d say we need to redefine ”scale” to mean a relevant number of users - nobody has a billion users, few have millions of users, most have thousands of users on a good day.
@mikaeleiman @collinsworth @bjoreman telcos in Asia have more than a billion users. They use #OpenLDAP...
@collinsworth @tychotithonus but if our architecture isn't webscale, how will we ride the hockey stick from zero customers to... umm... zero?
@collinsworth so true. A very real requirement in some markets/use cases is the ability to scale *down* and the kind of highly distributed solutions which work at Amazon scale just do not do small scale efficiently
@collinsworth what Amazon and Google won't tell you is they run all their own infrastructure on #OpenLDAP. And it doesn't need armies of engineers to optimize because it already scales perfectly linearly across arbitrarily many CPUs without any tuning.

@collinsworth
Yes, and there will be examples of exactly this, scaling without infrastructure costs - so a single part time person can create an app that scales world wide without spending a cent on hosting.

#Autonomi is a secure, privacy focused platform that will change the rules, expectations and possibilities in favour of developers and users.

@happyborg ouch, someone picked an unsearchable product name there 😬
@collinsworth ...back in 2016, and after over two decades of working for big telco names, we decided to go 'solo' and put all that wireless exp to good use...omg what a wild ride of thinking all was gonna be a 'piece of cloud cake' (it wasn't, nor it is reliable or cost-effective), with dozens of proprietary solutions tried-n-tested TO having no alternative but to do ALL ourselves, in-house with the strictest of budgets, open-source, non-proprietary tools - and the results are just beautiful
@collinsworth So much *this*. Small, self–contained binaries that store all their state in a SQLite database, where you backup those two files onto a removable disk you physically put somewhere safe.
@collinsworth the ideal usecase is a practice-oriented business running out of a server a nephew set up ten years ago before going off to college and forgetting all about it
@collinsworth My favourite philosophy for these tech stack choice questions is "Choose Boring Technology" https://boringtechnology.club/ Similar to what you're saying, but it's not exactly about targeting the "cheap-ass VPS somewhere". If you want room to grow that may be a mistake (yes, maybe you have some cringeworthy optimism about having 1000 times the number of users some time soon) but then again the well-established well-used boring technologies are usually good at scaling down as well as up.
Choose Boring Technology

@harry_wood @collinsworth

Definitely. I used some of the hot new cloud-native projects in the OpenID connect / identity aware proxy / identity management space.

Some had glaringly obvious security bugs that remained undetected for months, only partial support for the required standards and a crappy documentation.

In the end it turned out apache with mod_openidc and keycloak were the boring but ultimately best (by far) solutions to the problem.

@collinsworth this is the way!

The inefficiencies in the industry is probably by design to generate more revenue out of training, cloud costs, etc.

@collinsworth @joel I would absolutely (pay to!) subscribe to that magazine.
@collinsworth The #NVDARemote project for the #NVDASR screen reader is a brilliant example of this.
JAWS, the only viable commercial competition to the #opensource NVDA can cost up to £2,100 for an employee to use it given the right (or rather wrong) conditions.
JAWS has a feature called Tandem to let you control another machine.
A £12,000 croudfunding campaign in 2015 provided an opensource
alternative to this control for NVDA users.
Given the number of servers I've had to spin up over the last decade at work, I'd have spent about half of that on JAWS licensing to do my job. This software sits there, running on the cheapest VPS's or smallest Pi Zeros. It took investment, but I and hundreds of blind people use it daily now.
@collinsworth ha yeah well look over here 😄
@collinsworth
I've worked at both huge-scale companies and small. Both require very competent people to avoid tech failure - although huge vs small require different competencies.
@collinsworth @dsalo there was a time when Amazon and Twitter were scrappy startups that had genuinely figured out ways to move rapidly and build lightweight scalable systems without “enterprise” baggage, and that time was 20+ years ago
@chronodm @collinsworth @dsalo This is the truth. Anyone who goes to work at one of these companies today might as well go to IBM or Oracle instead. Amazon and Google haven't been providing any opportunities in innovation for 99% of their employees for decades at this point.
@collinsworth Small things that can run forever on a small box in someone's cupboard make me happy.
@collinsworth This (ancient) piece on a dating site run by one person from about a single rack for about 6 billion pageviews a month is memorable for me.
https://highscalability.com/plentyoffish-architecture/
PlentyOfFish Architecture - High Scalability -

Update 5 : PlentyOfFish Update - 6 Billion Pageviews And 32 Billion Images A Month Update 4 : Je...

High Scalability
@collinsworth and some jamstacks were too efficient to be profitable
@zachleat @collinsworth I feel like I'm gonna spend my whole career wondering if, instead of making supply more complex/expensive, we could somehow grow demand for cheap simple things

@eeeps @collinsworth I remain hopeful that the answer is yes, for a typical relationship of efficiency <=> profitability!

VC funding complicates this—especially when an inefficient thing is more profitable. Incentives!

@collinsworth hadoop and kubernetes have certainly been great technology for staff engineers. so many years of job security.
@collinsworth this was essentially my message from my PerfNow talk: https://youtu.be/f5felHJiACE?si=eCLusFP4iWa-_BRw – I included how my teams were burned by this, so somewhat of a case study.
The Unbearable Weight of Massive JavaScript | Ryan Townsend | performance.now() 2023

YouTube
@collinsworth this is a big part of the book I'm working on, at least as far as math is concerned https://pmfpbook.org
My next book will be "Practical Math for Programmers"

tl;dr: I’m writing a new book, sign up for the announcements mailing list. I’ve written exactly zero new technical blog posts this year because I’ve been spending all my writing efforts on my next book, Practical Math for Programmers (PMFP, subtitle: A Tour of Mathematics in Production Software). I’ve written a little bit about it in my newsletter, Halfspace. There I rant, critique, brainstorm, and wax poetic about math and software.

Math ∩ Programming

@collinsworth
I think people use shortcuts when evaluating tech that's similar to how bikeshedding works.

Compare frameworks on benchmarks only rather than how they help structure applications, as the latter evaluation is much harder.

Use something that is used by big tech because they surely have thought it through and our tiny app is a temporarily embarrassed web scale app, right?

Nobody got fired for buying IBM is also a classic.

@collinsworth yes, that's extremely important. But then, the hard question is : how do you identify "what works" when you don't have the vanity proxy of "it is now a billions dollar company"? It's much harder, atomized.
Plus, often these kind of companies really don't have a public relation channel (perhaps b/c of there focus on other things).

That being said : I co-run rudder.io since ~15 years. It's a small company, but we're profitable since year 1, the product is appreciated, ND we're still able to massively evolve it for the better. Ford it qualifies for "tech that works without a army of staff eng"?

@collinsworth
If you're talking about "per year" and US salaries, half a million dollars gets you maybe 3 people, not a small army.
@collinsworth Check out #Haxe, my favourite languages, transpiles everywhere and built entirely by enthusiasts.
@collinsworth we're literally this case. i get annoyed when i see people (mostly DHH) shit on serverless tech or the cloud, saying it's cheaper to run your own data center. Not when you're a small agile tech team with tight budgets, it's not.
@collinsworth Agree, but the incentives to find such tech are just not there.
@collinsworth I regularly say something similar, which is "why are you trying to imitate a company that literally has more money than it knows what to do with"?

This also looks “so frontend development.”

For 15–20 years we’ve been looking at how big companies are solving their big problems, instead of learning how those of us with small problems (individuals and SMBs) best solve their small problems.

That’s our Occam’s razor why frontend devs know Bootcamp/Tailwind, jQuery/React, bundle/deploy, and have every 10-visitor site sit on all edge servers—but for whom HTML is `div` and CSS is broken.

The craft died in Big Tech.

#frontend #html #css #craft

@collinsworth My current hobby-horse is the 99%+ waste in how Apple and Amazon currently handle podcast feeds. 1 engineer day each and some thought could eliminate most of that without breaking the the tricky corner cases that they certainly deal with. And would lighten bandwidth and CPU load on 100k+ sites (and the Internet) at a guess. But no glamour in that work! No case study, unless to burnish their green credentials.

https://www.earth.org.uk/RSS-efficiency.html

#RSS #Amazon #Apple #podcast #efficiency

RSS Podcast Feed Inefficiency

Climate cost of bad feed handling #RSS #podcast #Podbean #Spotify #Google #Apple

@collinsworth THIS THIS THIS.

Honestly, it was incredibly dispiriting to hear the words "we don't know if big websites will use this feature so arent sure we should develop it" from a member of the Safari team this week.

Like... can we please stop developing web tech *specifically for giant sites and corporate entities*. That is explicitly *not* what the web was built for.

It feels like such a betrayal.

@collinsworth @ash you can do a lot by picking boring technologies but all the cool blog posts on ci systems or whatever are written by people with tools teams.

Those boring tools lock you into other decisions. Use GitHub? You’re probably locked into the GitHub issue tracker as well (unless you want to do integration, so you need a tools team). And their ci. Etc.

And this is _good_. These tools scale cheaply because you make fewer decisions. But it makes them hard to adopt incrementally.

@collinsworth
I don’t think this is an accident if you look at the proliferation of SaaS and PaaS startups out there. The whole VC funding model is based on the premise that if you aren’t building FANG scale infrastructure you aren’t a serious company. Mostly because investors are also invested in keeping that hype cycle propped up.

It’s ludicrous tho - I run infra at an enterprise SaaS company on a handful of VMs and DBs, and skeleton crew of software developers. You just need to avoid the hype and have a leadership team disciplined on being cost efficient.
@collinsworth but there are - the high availability blog for example (used to?) have tons of those and if you look into a lot of scaling articles they also show how amazing lots of tech is in the hands of a tiny team of 2-4 people